A key to this "mystery" are the Amalekites.
These elite desert warriors attacked the Jews as soon as they came out of Egypt, and were a thorn in their side right up until the very last one (Haman) was killed in Esther's time.
And God removed the memory of them from off the face of the earth.
Exodus 17:14 And the LORD said unto Moses, Write this for a memorial in a book, and rehearse it in the ears of Joshua: for I will utterly put out the remembrance of Amalek from under heaven.
Evidently God removed evidence of the Jews in the wilderness also, when He cleaned up on the Amalekites.
Another thing to keep in mind is the fact that the Jews didn't even have reason to leave any clothes behind -- not even their shoes.
Nothing wore out.
Deuteronomy 29:5 And I have led you forty years in the wilderness: your clothes are not waxen old upon you, and thy shoe is not waxen old upon thy foot.
In addition, the Israelites were not haphazard and sloppy in their goings forth.
Numbers 20:17 Let us pass, I pray thee, through thy country: we will not pass through the fields, or through the vineyards, neither will we drink of the water of the wells: we will go by the king's high way, we will not turn to the right hand nor to the left, until we have passed thy borders.
Numbers 20:18 And Edom said unto him, Thou shalt not pass by me, lest I come out against thee with the sword.
Numbers 20:19 And the children of Israel said unto him, We will go by the high way: and if I and my cattle drink of thy water, then I will pay for it: I will only, without doing any thing else, go through on my feet.