Hmmm, I thought that baby's weren't supposed to have milk until 1 year to prevent allergies.... I gave my son banana flavored ceral at 2 months old (doc said he could have ceral) and he now has banana allergies (what was I thinking???) Anyways, I would think that formula would have more nutrients so my OPINION would be to stick with formula. You have to decide for yourself if you feel like he'd be getting enough nutrition with whole milk.
HB
The research I'd been working on (long before I was pregnant) doesn't show anything as far as milk causing the allergies unless they've already got issues with milk casein...
However, it does say to hold off on cereals (grains etc) until after they're a year old - and same for some fruits and nuts... and definitely with honey. I know one thing, that will definitely be difficult for us.

But in so doing, it will give our kids a better chance at not having Celiac Disease like me, unless it's already programmed in their genes and they're more prone. (at this point they have a 50/50 chance)
I did find that untreated milk or solely pasteurized but not homogenized (what I drink) is better esp if watered down. (I just found
this one online recently)
Any more they put additives in many of the formulas such as DHA, ARA, or BHT, aspartic acid, glutamic acid (this is MSG - I am allergic to this) , soy, corn and the like, which in the long run are far worse. - so using formula would create an issue when having to stave off of the grains (incl soy and corn) and it's in the formula already.
I'd definitely seek a knowledgeable Pediatritian and lay it all on the table, every concern you have, the research you've done and your health concerns/issues before switching to either.
Water is always a good addition. People thought my mom was crazy for giving us a bottle of water to drink, or watering down any juice we'd have (like 10% juice and 90%water) so she could see if we had allergies and also stave off problems such as diabetes (since that runs in the family on both sides) or other sugar issues... but I find now that many Peds are reccommending it.