I try not to let it worry me too much anymore.
Just keep reminding yourself that your relationship is with God not with an institution, and God is everywhere. Churches are good places to meet and fellowship with other Christians, however you don't have to "join" or "swear allegience" to a Church organisation in order to commune with God. Most importantly you shouldn't feel pressured to attend or be under threat of being ostracised if you fail to attend.
Christianity is all about free will. The free will to accept or reject Christ's free gift of salvation through his sacrifice. The free will to follow Christ to the best of your ability. And so it is with free will we attend church to fellowship together.
To attend church for any other reason but freely and by ones own choice is far from Christ indeed. The one thing that sets Christianity apart from all other religions is that it is not a religion that forces followers to believe or attend gatherings whether they like it or not. The beauty of this, is that invariably attendees at any Christian church on any given Sunday are there by choice not by "law", and this results in a wonderful atmosphere to worship and fellowship.
Hebrews 10:25
25 Let us not give up meeting together, as some are in the habit of doing, but let us encourage one anotherand all the more as you see the Day approaching.
In Hebrews 10:25 we are encouraged to meet together regularly for encouragement, not on a certain day of the week or in a certain manner, or that it must have a certain hierarchical structure. Simply that we not fail to keep in touch and encourage each other in our "walk".
It is man who has built a whole "industry" out of Hebrews 10:25, and in some instances gone as far as to make attendance compulsory and even viewing non-attendance as sinful.
I know it is a bit of a cliche, however, "Being a Christian is 24/7 existance not just 1 hour on Sunday" The real Church, we the people, are out in the world every minute of every day, this is where God's work is done, not in a building on Sunday for an hour or two. So as such we shouldn't get our knickers in a knot about this church or organisation or that church or organisation, because when all is said and done, it should have very little bearing on your personal relationship with God, except to provide fellowship and encouragement with like-minded Christians.
I think too many people are in love with church and religion, and forget to love God and one another. When this happens, then a church ceases to become a place of worship and becomes an institution.
Having said that, don't give up, there are plenty of great Christian Churches out there, that are true places of meeting, encouragement, fellowship and corporate worship. Which don't come with unrealistic man made rules, restrictions, superficial window dressing or unnecessary baggage.
If I were to paraphrase the New Testament, I would say;
"Love God and Love your neighbour as youself, and encourage one another as you strive to follow Christ."
Its all about love.
Bless Ya