Tertullian, who was earlier than Augustine and others that you quote, in his work On Modesty makes the following point in chapter XXI, saying
"In Peter himself the Church was reared; that is, through Peter himself; Peter himself used the key; you see what key: 'Men of Israel, let what I say...