I have felt this way, too. I sometimes have to really make an effort not to say something that I might regret to people who are being unfair. I am very opinionated, and it's hard for me sometimes to refrain from exrpessing exactly how I feel when these issues come up.
For me, what most upsets me is when it concerns my children. I have heard (not directly, of course) that a few people have said that they wished I wouldn't come to certain events because my kids might be too noisy. Of course, they don't mention any of the other families with young kids because they are just thrilled when any of them attend church programs. However, if my kids are not perfect, then they don't want us around. And I will do everything I can to protect my kids from hearing comments like that. We have rules, and we expect them to obey us, but we do not expect them to be perfect just because they are the pastor's kids.
The other thing that really annoys me is when people criticize my husband unfairly--for instance, when he takes a stand on principle and makes a decision that someone doesn't like, which they take personally and then proceed to gossip about it with all of their relatives and friends.
Anyway, back to your original question, what I have had to realize is that I can't change other people. No matter how many sermons we do or Bible studies we lead or whatever, there are some people that just never seem to get the point of being a Christian. All I can do for some of them is pray and try to be a good example myself.
The only person that I can change is myself. I tend to be very critical of myself and very perfectionistic, so I don't even need the church members to do it for me. I have had to learn gradually (and I'm still working on it) to lighten up on myself and not obsess about all the things that I could have done differently in the past--something that is difficult when others are adding to the criticism. I think that it is important to be realistic about the things that you can and can't do and to stick to your convictions, no matter what other people say. If I am walking with God and relying on Him to help me do the things that He wants me to do, then I have an assurance that is not so easy for others to break down.
Another thing that's really hard for me to do, but which I think is important, is to be kind even to those who are being unkind. I was reminded one day recently of the text in Proverbs that says, "A gentle answer turns away wrath, but a harsh word stirs up anger" (15:1). I had received a rather impatient e-mail from a church member who was irritated with me for not finishing something as soon as she would have liked it done. I was kind of hurt by it at first because I thought she was being a bit selfish, complaining about how busy she is and how much I was inconveniencing her. When I mentioned it to my husband, he said, "Maybe if you give her a gracious response, she won't be annoyed about it anymore." So I wrote her a brief but nice e-mail. Then, when I saw her in church the next week, she came over and gave me a hug and was very friendly. I found out later that she had had an especially busy week at her job and was under a lot of stress, so I thought that probably affected the way she wrote to me. I was glad that I had not followed my first impulse, which was to write back a similarly annoyed message.
Another thing that I struggle with is making friends in the church. I feel this wall of separation that I can't cross because I'm "the pastor's wife." Most of them don't seem to see me as a friend or even as a regular person. They all have their own friends already, and it's hard to fit into their little social circles. I can't talk to most of our church members about personal things because they gossip too much. Obviously, I can't talk to them about problems that I have with specific people. I have to keep a lot of it inside, and it gets to me after a while. I guess that's where prayer comes in. I find it easier to deal with all of the stress and unrealistic expectations--and everything in life, really--when I make more time for prayer. It's especially difficult to harbor bitterness or anger toward other people when I pray for them.
On a practical level, I have been getting more involved in activities outside my own church to make friends. I go to a non-denominational women's Bible study/mothers of preschoolers-type group (where they provide child care). Also, I have been thinking of trying to start some kind of monthly informal get-together (like for lunch) with the other pastor's wives in town. The men have ministerial association meetings, but I thought it would be a good idea for the women to get together regularly, also, because we can share some of these frustrations that we all experience.