My belief is that the church consists of the believers. The church is not the building they go to. We can, and should, worship anywhere at any time.
The analogy I think of is coals in a fire. Huddled together, each coal warms the others, and the fire keeps burning. Separated, the spark will die out and the coals will go cold. This is why we do need to meet periodically with other believers to pray, study His word, take communion, and other acts of worship. When/where we do so isn't the most important thing, as long as we don't lose our heat.
The argument has also been made that it is never necessary to meet with other Christians in order to be a Christian. Well, granted, you're still a Christian as long as you're washed in the blood, even if you never in your life have fellowship or worship as a part of a congregation. However, the counterargument is this: Jesus is the head of the church, and the church is His body. How can one be a part of the head, without also being part of the body?