Ugh, I hate this kind of argument.
Yes, there was a time - perhaps - when clergy spent four days a week visiting, one writing a sermon, and one taking services (with one day off). That was also a time when what was expected of clergy was very different, and the tools available to do it were very different.
I can't administrate the parish, manage compliance with a raft of different legal obligations, deliver in-depth teaching programs to people of every age from newborn to 90-something, lead creative and vibrant worship midweek and on Sundays and in nursing homes, spearhead all sorts of community building and outreach activities, and manage more than a couple of pastoral visits a week, who are likely to be folks in particular crisis. (In fact, I can't manage all of that anyway, but that's a separate issue).
Also, technology has changed. Today pastoral care might mean that I scroll through my FB feed, notice that a parishioner has posted about having skin cancer, pick up the phone and ring and ask how they are. I'm not visiting, but that doesn't mean I'm out of touch.
More on topic: no, I don't believe that clergy are, by default, closer to God than laity. There are many incredibly faithful, prayerful, profoundly holy people who are not called to ordained ministry. I would hope, at least, that indifference to God is not really an option for the clergy.