If you are 'in Christ', you are the body of Christ since the church is the body of Christ. Jesus is talking about faith, have you read the context of the passage?
Aware that his disciples were grumbling about this, Jesus said to them, “Does this offend you? Then what if you see the Son of Man ascend to where he was before! The Spirit gives life; the flesh counts for nothing. The words I have spoken to you—they are full of the Spirit and life. Yet there are some of you who do not believe.” For Jesus had known from the beginning which of them did not believe and who would betray him. (John 6:61-64)
'Yet there are some of you who do not believe', did you catch that? Jesus said he was the bread of life, that doesn't mean he thought himself a loaf of bread. The communion of the Lord's Supper is very much an important time of fellowship with God and other believers, I would never try to diminish that. What God cares about isn't so much what rites and rituals we perform, that doesn't mean they are not important. To be in the faith is to be in Christ, to abandon the faith is to abandon Christ. What's more, those who stayed, stayed because of Christ:
Simon Peter answered him, “Lord, to whom shall we go? You have the words of eternal life. We have come to believe and to know that you are the Holy One of God.” (John 6:68-69)
Grace and peace,
Mark