Quite honestly, hermits themselves still had interaction with their Church community. They still received the Sacraments, and to do so required them to seek out the Church. Hermits are those who lived on their own, so that they could focus on prayer, but they didn't survive on their own. There is a reason why there are many hermits who are Saints, they were known and loved by those who knew them. To be known and loved requires one to have some interaction with the outside world.
Think about it. If someone went out into the mountains to focus upon prayer, and reached a very high level of sanctity, even to the point of being able to perform miracles; if no one knows he is there, then how would he be known? Its like the phrase, "if a tree falls in the woods, and no one is there to hear it, did it make a noise?" The answer is obviously yes, but then who cares?