You know it couldn't hurt to expand your horizons and go out with them a couple times. You don't have to drink like a fish, heck Jesus drank whine, you can have a class or two of something just to be friendly. The point of going out it not for happy hour, but to talk to these coworkers, hang out, get to know them. So when you tell them no you tell them they are not worth getting to know so it can be taken offensivly.
I'm not saying you must go out with them all the time, or drop everything and go with them. Just if you have time, if its convenient sometime I think you should. There is a wide and wonderful array of hobbies in this world. It is foolish to limit yourself to just a couple.
A lot of times when i go out, I will play pool, or darts, or video games, it doesn't have to be about sitting around and getting drunk. You can always just order food from the bar too and have a bite to eat, the point is to just be social.
If you alienate yourself from your coworkers you can not expect to not be alienated from them. If you still desire to stay far away from their hobby you could always at least invite them to do something with you.
Well I'm praying for you and your co-workers,
Good luck buddy,
BEN