You don't trust man, you trust Jesus and His Holy Spirit. And you test out what is being said by God's Word, no matter who it is. And guess what? People can make up things, we ourselves do it too. This is the battle of the flesh we have to kill everyday. And the same mercy that God gave you whenever you mess up, we are commanded to give to others just as much. Not saying that it justified in what they said to you, however they are not your enemies, the Devil is.Don't make them that. In all that you said at least from just what you're telling us, you've consulted the pastor, the people in the church, your friend, but I don't hear you mention about you consulting Jesus on it.Try to make Him your first go-to on things.
Christian community is much more bigger than just the people in your community, so it wouldn't be wise to start to discourage yourself from being in one. There is such a thing as the "wheat and the tares"among our community (Matthew 13) and God will handle it.We still need each other to sharpen each other in the faith and hold each other in prayer, because the World will NOT do that.