The difference is Christ foretold the destruction of the temple. If Christ says the temple will be destroyed, then His word be done. If Christ says his Church will prevail of Hell, then his word be done.
Also, where is your reference to Christ stating that he meant the church as an ideal? I see no evidence for that in the Greek or from any of the church fathers. In fact just two chapters after Christ gives Peter the keys, Christ says:
"If he refuses to listen to them, tell it to the church; and if he refuses to listen even to the church, let him be to you as a Gentile or a tax collector" Matthew 18:17
Christ refers to the church as an institution. A concrete, real, organization. Not an ideal. And this makes sense because he had just told Peter that the church would be built upon Peter.
So again I ask, what evidence do you have that Christ refers to the Church as an ideal in Matt 16?