This was an interesting podcast episode. It says that evidence that Peter was the first pope is in part because Saint Peter’s Basilica is so very large compared to other tombs such as that of Paul etc
There's little reason to doubt that the episcopate in Rome can be traced back to St. Peter; just as the episcopate in Antioch is traced back to both Peter and Paul; and the episcopate in Jerusalem goes back to St. James (etc.).
The issue of the Papacy, however, is very different. That Peter was the first bishop of Rome and that the bishop of Rome sits in St. Peter's chair is rooted in Christian history. We even can see how, very often, in the ancient Church the bishops of Rome were highly respected as defenders of Christian orthodoxy, and often highly honored. What we lack, in the ancient record of the Church, is any notion of any bishop having anything resembling a universal authority over the whole Church. In fact we actually have, from the words of those men who have sat in St. Peter's chair, statements to the contrary. St. Gregory the Great, when John IV of Constantinople tried to claim a position of headship over the whole Church, fiercely criticized him, and went so far as to say that any man who would try to claim be a universal pastor over the Church would be like a forerunner to the Antichrist. The ancient Canons of the first Ecumenical Councils are clear that while Rome is highly honored, the dioceses of Constantinople, Alexandria, Antioch, and Jerusalem are also highly honored (this is the root of what is known as the historic Pentarchy).
The assumption of greater ecclesiastical, and even political, power are later innovations that entered into the Western Church, largely after much of the political power of the East had been broken, and there was a growing schism between East and West. The beginnings of this schism show up at least as early as the Photian Controversy in the 9th century, but was truly seen in the Great Schism dated to 1054 over the Filioque Controversy; and was effectively completed by the time of the Council of Florence in the in the 15th century. After the Western Schism (aka the Papal Schism, in which there was at one point three competing claims to the Papal throne) there was a fairly rapid accumulation of Papal authority, resulting in the death of the Conciliar Movement, a reinvigoration of Papal political and ecclesiastical authority, and which has continued into modern times (e.g. the First Vatican Council in the 19th century which established Papal Infallibility as dogma).
That the bishops of Rome are the successors of St. Peter and sit in his apostolic chair isn't the controversy. It's all the unique powers and particular claims about the supremacy of St. Peter's Chair which came much later, and have evolved and grown over the centuries, that are the heart of the controversy. This is what both the Eastern Churches and the mainstream Churches of the Reformation take issue with. That the Papacy is by its very nature contrary to the faith and practice of the Historic Catholic Church of Jesus Christ. The Church of Jesus Christ only has one Head, and that's Jesus Christ Himself, who rules and reigns at the right hand of the Father, and over His whole Church on earth and in heaven as Lord, King, and God. He is her Shepherd, He is her Foundation and Rock. No bishop, no pastor, can claim to be what belongs solely to Christ: The universal Headship of the one, holy, catholic, and apostolic Church.
-CryptoLutheran