Well a woman might not want to have sex for multiple reasons. Hormonal changes and fluctuations may alter her libido, stress and fatigue can be real mood killers, her partner may forget to give her TLC and so make her feel like her needs are not being met, her body may go through PH changes that make sex painful, she might be on her period and not tell her husband, or she might just not feel like doing anything more than holding and kissing.
And guess what, these same issues (with the exception of the period) affect men.
It is a mans fault if he has an affair. His wife is not forcing him to turn to another woman for sex, since there are ways to relieve yourself that don't require breaking the marriage vows you made before GOD to love and to cherish each other through thick and thin. I know a young man who was married for 8 years, and of those 8 years 2 of them were spent in a sexless marriage, and he stayed faithful to her during those years. Eventually he got a divorce because there was alot of bitterness and anger in the relationship that just wasn't working itself out, but not having sex is NOT a valid reason to have an affair, and it is possible to resist that temptation.
Just because you're married does not automatically give you free access to their body. Also, the belief that if you go ahead with it, even if you're not in the mood, will put you in the mood doesn't always hold true. Sometimes it just makes things awkward.