Gentle Lamb, it would be wrong to "bail" on your church in a moment of anger. It would also be wrong to do so if you had done absolutely nothing to try to moderate or correct the situation, such as give whatever is wrong a little time to settle down or else to say something to the pastor or a board member.
From what you told us, however, you are in synch with all of that and are facing the very real prospects of the situation never improving.
If that's the case, do not be reluctant to seek out a better congregation.
Your perception is correct--staying in a congregation like yours is at present but which could fairly easily be replaced (similar doctrine, worship style, denomination, etc.) only serves to damage your walk with the Lord. I know people who are at odds with their church for some very good reasons, but they will not leave because "no one can chase me away" or "my family helped build this church" or something else like that...and they come home from every Sunday service simply boiling inside. There's nothing good about that.
You are not a quitter if, under the circumstances I think exist in your current congregation, you start looking around.