There is more that indicate they were not there and only the shepherds were.
After the Wise men visited Herod and made there way to see the new "king' they left after agreeing they would return. I imagine Herod would have given them some time to get to the childs location. Herod waited some time (1 month, 2, 3...? who knows,,) and then realised they were not returning. He sent out his henchmen to kill all children under 2 years of age. I dont think Herod would have waited 2 years before realising they had not returned. Herod seemed to think the baby was already born when the star appeared and that the Magi had been travelling some time just to reach him and still had further to go to see the child.
Scriptures:
7Then Herod called the Magi secretly and found out from them the exact time the star had appeared. 8He sent them to Bethlehem and said, "Go and make a careful search for the child. As soon as you find him, report to me, so that I too may go and worship him."
9After they had heard the king, they went on their way, and the star they had seen in the east[5] went ahead of them until it stopped over the place where the child was. 10When they saw the star, they were overjoyed. 11On coming to the house, they saw the child with his mother Mary, and they bowed down and worshiped him. Then they opened their treasures and presented him with gifts of gold and of incense and of myrrh. 12And having been warned in a dream not to go back to Herod, they returned to their country by another route. 13When they had gone, an angel of the Lord appeared to Joseph in a dream. "Get up," he said, "take the child and his mother and escape to Egypt. Stay there until I tell you, for Herod is going to search for the child to kill him." 14So he got up, took the child and his mother during the night and left for Egypt, 15where he stayed until the death of Herod. And so was fulfilled what the Lord had said through the prophet: "Out of Egypt I called my son."[6]
16When Herod realized that he had been outwitted by the Magi, he was furious, and he gave orders to kill all the boys in Bethlehem and its vicinity who were two years old and under, in accordance with the time he had learned from the Magi.