"It seems to me that the most likely scenario is that Jesus was born around 6 B.C., the wise men (by their own account) arrived in Jerusalem two years later in 4 B.C., and in that same year Joseph, Mary, and Jesus fled to Egypt, Herod died, and they returned."
There is nothing about arriving two years later. They saw the star which meant Jesus was born and came to worship him. There is no reason to believe it took two years to travel there.
Mat 2:1 Now when Jesus was born in Bethlehem of Judaea in the days of Herod the king, behold, there came wise men from the east to Jerusalem,
Mat 2:2 Saying, Where is he that is born King of the Jews? for we have seen his star in the east, and are come to worship him.
Mat 2:3 When Herod the king had heard these things, he was troubled, and all Jerusalem with him.
Mat 2:4 And when he had gathered all the chief priests and scribes of the people together, he demanded of them where Christ should be born.
Mat 2:5 And they said unto him, In Bethlehem of Judaea: for thus it is written by the prophet,
Mat 2:6 And thou Bethlehem, in the land of Juda, art not the least among the princes of Juda: for out of thee shall come a Governor, that shall rule my people Israel.
Mat 2:7 Then Herod, when he had privily called the wise men, enquired of them diligently what time the star appeared.
Mat 2:8 And he sent them to Bethlehem, and said, Go and search diligently for the young child; and when ye have found him, bring me word again, that I may come and worship him also.
Mat 2:9 When they had heard the king, they departed; and, lo, the star, which they saw in the east, went before them, till it came and stood over where the young child was.
Mat 2:10 When they saw the star, they rejoiced with exceeding great joy.
Mat 2:11 And when they were come into the house, they saw the young child with Mary his mother, and fell down, and worshipped him: and when they had opened their treasures, they presented unto him gifts; gold, and frankincense, and myrrh.
Herod killed children ages 2 and under but he didn't know when Christ was born. The fact that he wanted newborns killed also shows he thought Christ was possibly newborn but made sure just in case to kill up to 2 years old. So that is children 1 day old on up to 2 years. If it took the magi two years to travel from the birth to arriving then Herod would not have needed to kill children under the age of 1 or 1 and a half since they would have been born after Christ.