I think it brings a sufficient amount of doubt on your claim, and since the traditional view is that the wise men came when Jesus was closer to 2 and tends to support that view. The wise men I might add went to a house not a staple so at the very least it shows that Mary and Joseph seemed to be established enough in Bethlehem that they moved out of the stable. How long exactly is too long in your view anyway? it gets a little silly because in the end what exactly are you defending anyway?
Matthew's account is the only account that brings the wise men up so should we not look to Matthew's account to ascertain when they arrived and how old Jesus was? So what is Mathew saying? Luke may very well be telling the same account but simply skipped over the wise men and said they went home but an exact timeline is not really what the point, he's just picking and choosing the moments he wants to tell.
If you want to look at the modern scholarly view the wise men and subsequent Herod's killing on the children under 2 is a myth to prop up Christ as the true Messiah. These sort of posthumous accounts are actually common in Eastern thinking as the value is not about if it's true or not but rather how well does it tell the truth. If the truth is Jesus is God's son then, of course, wise men came to visit him, and of course, he is a typology of Moses as these things declare the truth with stronger notes. If it really happened or not is the point, the point is Jesus is important and we should stop and take notice.
Take a look at the Catholic church, Peter is declared the first Pope, but is he actually functioning as the Pope? no of course not, nor was that his role, but it doesn't matter, he is posthumously given the title of Pope so that the office of the Pope has a single line of succession and is given more authority. Those outside the Catholic church disagree but those inside it vehemently defend it. We, however, all agree with the account of the wise men (at least that they came) regardless of what evidence seems to suggest because we believe in its goal that Jesus is the Son of God. Since the wise men account agrees with this we accept it as fact and don't care about others information says. It is a religious blindness and you're taking the account too literal with information that really can't be reconciled.