It goes along the lines of gender roles, and also (at least here in the US) the propensity for divorcing moms to get domiciliary custody of the kids. Thus, a daddy alone with his kids is easily suspect of all sorts of crazy assumptions.
My husband was a single dad (sole custody of both of his cuties) for over six years, and you would not believe how many people initially assumed that his wife had died! It boggled people's minds how a mother could leave her children, and the father take them on, alone, willingly...To them, it just wasn't "normal".
After we were married, when hubby injured his knee, he stayed at home with the kids, and I worked. There were quite a few people, even within the church(!), who wrinkled their noses at the arrangement...Like my husband was some sort of a deadbeat! First off, it's nobody's business but those in that sort of domestic arrangement, secondly, his knee was horribly injured to an extent that he couldn't walk, and thirdly, don't people have something better to talk about than a daddy who cares enough to actually parent his children?
A SAHD definately has my sympathy and respect. Not only does he do the work that a woman typically does, but he also endures the curiosity, scorn and prejudices from others who either don't understand the arrangement, or are too narrow-minded to do more than gossip about it. We've lived enough in the SAHD arrangement that I have a lower opinion of men who work all the time and neglect their family for the almighty dollar (and therefore isn't home for his kids at all), than of a man who doesn't work and tends his family diligently while the wife works.
My dream is to get enough experience and earning power so that my husband can stay home with the kids full-time again. Why? He's an excellent cook, he's incredibly efficient with the housecleaning, he ROCKS as a dad, and he's not afraid to get with his kids and play with them. He's sensitive enough to kiss a boo-boo or give a warm hug, and masculine enough to cut the grass, fix the van or change the oil. Also, I'm more temperamentally suited to the workforce than he is, and I have a higher potential both for earnings, and for continued education in my chosen field.
God gives us each the gifts to "live life more abundant". Sometimes, a man can have a gift that isn't suited to the sexual stereotypes of today's world.