Hi there,
This struggle deals with an issue I've had to come to terms with personally... whether to leave a church that you have invested many aspects of your life into or not.
Here's what you may want to consider: have you sacrificed a lot in other situations or relationships and felt guilty parting with them in the past? If so, why have you felt guilty? Explore any other feelings that you're experiencing due to this dilemma.
Sometimes, as Christians, we feel guilty about a lot of things that God doesn't want us to feel that way about. Leaving a church or even a relationship or friendship is one such thing. We feel like we have to sacrifice or "be a good Christian" and rescue something or someone that we have invested into. Sometimes, God wants us to come to the rescue. Other times, He is using the separation to lead us in a new direction.
Ask yourself, is the new direction I want to go going to benefit people for His glory? If so, maybe it's not such a bad decision. Ask yourself, is the church I'm a part of going in a direction that will glorify God? If so, maybe you can re-evaluate your part in that ministry and decide from there.
Like others have said, it won't be easy. I recently left a church that I had been a part of since I was 4 years old (and I'm in my mid 20s now), and it was one of the hardest decisions I had to make. The situation is different, but the guilt and heartache was similar. I had fears that people would look down on me for the change, and still feel a little guilty for not visiting. But recently, I realized that I just fit better with my new church. My old church was big and loud and I had never met the senior pastors (and that's great for some) and my new church is smaller and newer and I know the pastors personally. I feel like God is refreshing me through the change.
Your situation may be different, but the decision process may also be similar... just think about how you would respond to a friend going through the same situation as you. Would you make them feel guilty, or urge them to follow the peace in their heart that's leading them one way or the other? Try doing the same for yourself that you would do for a close friend. I hope that helps.