Let's go through this a bit to see something. Red is from your source in the OP.
"
1. We teach and declare that, according to the gospel evidence, a primacy of jurisdiction over the whole Church of God was immediately and directly promised to the blessed apostle Peter and conferred on him by Christ the lord.
2. It was to Simon alone, to whom he had already said: "You shall be called Cephas" [42], that the Lord, after his confession, "You are the Christ, the son of the living God," spoke these words: "Blessed are you, Simon Bar-Jona. For flesh and blood has not revealed this to you, but my Father who is in heaven. And I tell you, you are Peter, and on this rock I will build my Church, and the gates of the underworld shall not prevail against it. I will give you the keys of the kingdom of heaven, and whatever you bind on earth shall be bound in heaven, and whatever you loose on earth shall be loosed in heaven" [43].
3. And it was to Peter alone that Jesus, after his resurrection, confided the jurisdiction of Supreme Pastor and ruler of his whole fold, saying: "Feed my lambs, feed my sheep" [44]. "
Feed my lambs. Peter got the message for he says 1 Pt. 5:1-2 The elders which are among you I exhort, who am also an elder, and a witness of the sufferings of Christ, and also a partaker of the glory that shall be revealed: Feed the flock of God which is among you, taking the oversight [thereof], not by constraint, but willingly; not for filthy lucre, but of a ready mind;
Okay. Everyone agree with Peter? He got the message and passed it on to the elders in Asia Minor. Can't be any doubt about that, right?
" 1. That which our lord Jesus Christ, the prince of shepherds and great shepherd of the sheep, established in the blessed apostle Peter, for the continual salvation and permanent benefit of the Church, must of necessity remain for ever, by Christ's authority, in the Church which, founded as it is upon a rock, will stand firm until the end of time [45].
2. For "no one can be in doubt, indeed it was known in every age that the holy and most blessed Peter, prince and head of the apostles, the pillar of faith and the foundation of the Catholic Church, received the keys of the kingdom from our lord Jesus Christ, the savior and redeemer of the human race, and that to this day and for ever he lives" and presides and "exercises judgment in his successors"
Which successors? Peter said the elders in Asia Minor.
" the bishops of the Holy Roman See, which he founded and consecrated with his blood [46]. "
Whoa. What just happened? Where'd that notion arise? Not from Peter we've seen, but from where?. Here's Irenaeus a couple hundred years later.
[we do this, I say,] by indicating that tradition derived from the apostles, of the very great, the very ancient, and universally known Church founded and organized at Rome by the two most glorious apostles, Peter and Paul; as also [by pointing out] the faith preached to men, which comes down to our time by means of the successions of the bishops.
AH, III, c3.
Tradition derived from. Not scripture, but tradition assumed. But let's also assume that tradition is true and accurate. Where does that leave Christians?
For it is a matter of necessity that every Church should agree with this Church, on account of its pre- eminent authority, inasmuch as the apostolical tradition has been preserved continuously by those [faithful men] who exist everywhere.
AH, III, c3.
A matter of necessity to agree with Rome. Why? Its pre-eminent authority. What does that mean? Whatever it means, Irenaeus qualifies it, INASMUCH AS the apostolic tradition has been preserved continuously by successors.
We look backwards in time. Obviously OO believe Rome fell away from apostolic tradition. Obviously EO believe the same. Obviously P believe the same. Restorationists believe the same. Now, this is not to imply anything more about those groups and whether their dogmas/doctrines line up with apostles. Like RC, they may or may not. The point is Irenaeus says, you best agree with Rome because of its preeminent authority.
So, if we ignore and suspend Peter's clearly given lineage from scripture and substitute a tradition of men, then we are left with a choice. Follow that tradition of men, but only inasmuch as the apostolic tradition has been preserved continuously. Has it? No one agrees with it, but Rome alone.
So, where could one turn then?
Irenaeus says he can list successors in all the Churchs, but lists two more successions for consideration.
But Polycarp also was not only instructed by apostles, and conversed with many who had seen Christ, but was also, by apostles in Asia, appointed bishop of the Church in Smyrna, whom I also saw in my early youth, for he tarried [on earth] a very long time, and, when a very old man, gloriously and most nobly suffering martyrdom, departed this life, having always taught the things which he had learned from the apostles, and which the Church has handed down, and which alone are true. To these things all the Asiatic Churches testify, as do also those men who have succeeded Polycarp down to the present time,—
There again is Peter's successors, handing down what was learned from apostles and which alone are true. No qualification. And lastly,
Then, again, the Church in Ephesus, founded by Paul, and having John remaining among them permanently until the times of Trajan, is a true witness of the tradition of the apostles.
Ephesus. To spiritualize it, Christian love, if all else fails. Return to your first love.
Peace.