Well, let me consider a couple cases.
Joe and Janey have been married for years, and have a child. Joe cheats on Janey with Barbara, who is single. We have a case of adultery.
Joe and Janey are both single and living next to each other, and have been romantically involved for years. They have a child who lives with Janey. Joe cheats on Janey with Barbara, who is single. Since nobody is married here, there is no adultery.
Since the second case isn't going to be illegal, I see no reason for the first case to be illegal, so I don't support criminalizing adultery. I wouldn't really oppose it either, but I would oppose stiff penalties for adultery.