I think our trust in the Catholic Church doesn't have to based on any specific people or popes. People come and go, are sinners, are imperfect, and are always changing. The times are always changing also. In our lives we all go through good times and bad times, so the church will also go through good and bad times.
But the foundation of the Catholic Church is unchanging. I think that foundation is Peter. "And so I say to you, you are Peter, and upon this rock I will build my church, and the gates of the netherworld shall not prevail against it. I will give you the keys to the kingdom of heaven. Whatever you bind on earth shall be bound in heaven; and whatever you loose on earth shall be loosed in heaven.”" (Matthew 16:18-19).
In one of my RCIA classes, the priest told us a story about how the remains of Peter were discovered. During WWII, a search began for the tomb of Peter. With the permission of the pope, workers began excavating beneath St. Peter's Basilica, and eventually a grave was found, marked "Petros eni" or "Peter within". Bone sets were found, and when anthropologists studied the bones, they found that they belonged to a man about 60 or 70 years old who was 5 feet 7 inches tall, and of robust constitution, which could easily describe Peter who was a fisherman. The bones were wrapped in extremely expensive purple and gold cloth, which would make sense if he were the first pope. But something interesting was that the bones from the feet were missing. There was a story that when Peter was martyred, he was crucified upside down because he declared he was unworthy to die in the same manner as the Lord. When Christians came to retrieve his body, they had little time to do so without being seen by guards, so they cut off his feet and took the rest of the body.
It is interesting to me that the Catholic Church is literally built on the rock of Peter.