I suggest looking up what happened a previous time the Jews tried to rebuild the temple in 363 ad. It wasn't pretty.
From Ecclesiastical History, Book V, Chapter XXII (Sozomen (ca. A.D.375-447)):
The Jews entered upon the undertaking, without reflecting that,
according to the prediction of the holy prophets, it could not be accomplished. They sought for the most skillful artisans, collected materials, cleared the ground, and entered so earnestly upon the task, that even the women carried heaps of earth, and brought their necklaces and other female ornaments towards defraying the expense. The emperor, the other pagans, and all the Jews, regarded every other undertaking as secondary in importance to this.
Although the pagans were not well-disposed towards the Jews, yet they assisted them in this enterprise, because they reckoned upon its ultimate success, and hoped by this means to falsify the prophecies of Christ. Besides this motive, the Jews themselves were impelled by the consideration that the time had arrived for rebuilding their temple. When they had removed the ruins of the former building, they dug up the ground and cleared away its foundation; it is said that on the following day when they were about to lay the first foundation,
a great earthquake occurred, and by the violent agitation of the earth, stones were thrown up from the depths, by which those of the Jews who were engaged in the work were wounded, as likewise those who were merely looking on. The houses and public porticos, near the site of the temple, in which they had diverted themselves,
were suddenly thrown down; many were caught thereby, some perished immediately, others were found half dead and mutilated of hands or legs, others were injured in other parts of the body. When God caused the earthquake to cease, the workmen who survived again returned to their task, partly because such was the edict of the emperor, and partly because they were themselves interested in the undertaking. Men often, in endeavoring to gratify their own passions, seek what is injurious to them, reject what would be truly advantageous, and are deluded-by the idea that nothing is really useful except what is agreeable to them. When once led astray by this error, they are no longer able to act in a manner conducive to their own interests, or to take warning by the calamities which are visited upon them.
The Jews, I believe, were just in this state; for,
instead of regarding this unexpected earthquake as a manifest indication that God was opposed to the re-erection of their temple, they proceeded to recommence the work. But all parties relate, that they had scarcely returned to the undertaking, when fire burst suddenly from the foundations of the temple, and consumed several of the workmen.
This fact is fearlessly stated, and believed by all; the only discrepancy in the narrative is that some maintain that flame burst from the interior of the temple, as the workmen were striving to force an entrance,
while others say that the fire proceeded directly from the earth. In whichever way the phenomenon might have occurred, it is equally wonderful. A more tangible and still more extraordinary prodigy ensued; suddenly the sign of the cross appeared spontaneously on the garments of the persons engaged in the undertaking. These crosses were disposed like stars, and appeared the work of art. Many were hence led to confess that Christ is God, and that the rebuilding of the temple was not pleasing to Him; others presented themselves in the church, were initiated, and besought Christ, with hymns and supplications, to pardon their transgression. If any one does not feel disposed to believe my narrative, let him go and be convinced by those who heard the facts I have related from the eyewitnesses of them, for they are still alive. Let him inquire, also, of the Jews and pagans who left the work in an incomplete state, or who, to speak more accurately, were able to commence it.
