What happened to ancient Jerusalem and the temple (and to the people that heard Jesus's warnings but didn't follow the instructions He gave ) was definitely physical (and, I believe, everlasting).
Indeed.
The naysayers are going to have to face the biblical reality of AD 70 as stated here:
Matthew 21:40-45
When the Lord therefore of the Vineyard comes, what will he do unto those husbandmen? They say unto him, He will miserably destroy those wicked men, and will let out his vineyard unto other husbandmen, which shall render him the fruits in their seasons. Jesus saith unto them, Did ye never read in the scriptures, The stone which the builders rejected, the same is become the head of the corner: this is the Lord's doing, and it is marvellous in our eyes? Therefore say I unto you, The kingdom of God shall be taken from you, and given to a nation bringing forth the fruits thereof. And whosoever shall fall on this stone shall be broken: but on whomsoever it shall fall, it will grind him to powder. And when the chief priests and Pharisees had heard his parables, they perceived that he spake of them.
This event (The coming of the Lord of the Vineyard) was what the apostles had in mind when they promised their disciples the following:
-------------
Hebrews 10:37
For yet a little while, and he that shall come will come, and will not tarry.
1 Peter 4:7
But the end of all things is at hand...the time is come that judgment must begin
James 5:3, 8-9
[you rich ones]... it is in the last days that you have stored up your treasure!...the coming of the Lord is near...behold, the Judge is standing right at the door.
Romans 13:11-12
salvation is nearer to us than when we believed. The night is almost gone, and the day is near.
1 John 2:18
Children, it is the last hour; and just as you heard that antichrist is coming, even now many antichrists have appeared--from this we know that it is the last hour. They went out from us
Revelation 1:1,3
The Revelation of Jesus Christ, which God gave Him to show to His bond-servants, the things which must soon take place...for the time is at hand.
Matthew 23:34-36; 24:34
I am sending you prophets and wise men and scribes; some of them you will kill and crucify, and some of them you will scourge in your synagogues, and persecute from city to city, so that upon you may fall the guilt of all the righteous blood shed on earth, from the blood of righteous Abel to the blood of Zechariah, the son of Berechiah, whom you murdered between the temple and the altar. Truly I say to you, all these things will come upon this generation...So, you too, when you see all these things, recognize that He is near, right at the door. Truly I say to you, this generation will not pass away until all these things take place
-------------
It's really quite simple. They expected the coming of the Lord of the Vineyard and the Stone to crush their enemies and give the kingdom to a new nation, the Church (Matt 21:40-45). They were not here speaking of the end of the Cosmos, but the end of the Old Testamental period and the dawn of the New Testamental age of Christ's Church and Kingdom.