I'm not as far along. I just finished chapter four- and realized I felt the same about Roger's feelings toward Hester as I did in high school. He doesn't blame her at all, he blames himself and the unknown Dimmesdale (I mean, unknown to him). His revenge is toward Dimmesdale and not Hester, which floored me when I was in high school and still impresses me.
I guess I can not totally hate Chillingworth because he has this magnanimity toward Hester, he figures she has enough punishment. There is something about Chillingworth that appeals to me- his explanation of why he wanted to marry Hester, or maybe his logic, I don't know. I just don't see him as a villain. Dimmesdale was always the villain in my mind- his cowardice disgusts me, his willingness to let Hester bear all the burden of their shared sin.
Anyway, I need to read more to see if I have any pity for Dimmesdale. Right now, all my pity is for Hester, Pearl, and Chillingworth.