Interjecting here, I don't think Peter deferred to James at all, although that is a standard Protestant/Orthodox line. Except they don't see him even deferring, they see James as running the council and making the key decision.
What often gets missed in that analysis are these two quite important things:
1) Peter alone had already made the decision (without the need of a council) to allow Gentiles into the Church without circumcision, and this was by a direct act of the Holy Spirit (read Acts 10). The OP mentions a special grace given to the office of the Pope. I would refer to it more as a charism. But we see it in Matthew 16 when the Father reveals to Peter who Jesus is. And we see it in Acts 10 when the Holy Spirit directs the encounter between Peter and Cornelius and Peter gets the message -- the Gentiles are not to be denied Baptism and are thus allowed into the Church. And this message was delivered solely through the Holy Spirit working through St. Peter. Therefore, the council of Jerusalem was not needed to determine a doctrinal direction -- the Holy Spirit had already taken care of that. But it was needed to preserve unity, because the Church was headed to a schism. Which brings us to James....
2) The other thing that gets missed is that James was the leader of the "circumcision party." It was his group that was going around following after St. Paul and telling his Gentile converts they had to be circumcised (Galatians 2:12). The purpose of the council wasn't to determine correct doctrine -- the Holy Spirit had already delivered that knowledge through St. Peter. The purpose of the council was to bring James back into alignment with the direction of Peter and avoid schism. So Peter does not 'defer' the decision to him -- the decision was already made. And that is why once Peter speaks at the council we are told there is silence -- the debate is over. All that is left is to see how James then reacts -- will he defy Peter or align himself with him. And we see James cite Peter and defers to his leadership and aligns himself with the office of Peter, and schism is avoided.