But then you run into the problem of someone being punished for nothing, and does God really punish someone for the rest of their life for something they didn't even do?
Consider a kindly and faithful woman who meets a man and falls in love with him, only she doesn't know that this man has a hidden violent side to him. They date, and this violent side of him never arises because she's not around him enough during dating.
They marry after, say, a couple years of dating and things go well the first year or two and they have a child. Things start falling apart fast; the man starts getting frustrated, perhaps he starts drinking or what-not and becomes violent towards her and their child. She tries for years to make it work, she tries prayer, she tries calming him down, she tries counseling, but she inevitably ends up divorcing him because of his violent behavior. Maybe he even spends a stint in jail because of domestic abuse.
So now she's divorced. If we take adultery as the only reason someone can divorce and re-marry, this poor woman who did nothing wrong in the matter is now punished the rest of her life. She can't re-marry (assuming we're only allowing for divorce because of adultery), and she's now forced to live on her own. Let's say she married at a young age and was a housewife. She's supposed to somehow get a job and support a child on her own?
Does God punish people who did nothing wrong?
This is why pastors say that domestic abuse could (but we have no proof of) be another "valid" reason for divorce. Perhaps domestic abuse didn't exist much back then, and that's why it wasn't mentioned. I don't know. It just doesn't seem like the character of God to punish someone in this situation by forcing them to remain single the rest of their lives. Even widows are allowed to re-marry (I've not read anything in the Bible that says a widow can't marry again).