Yes. But it's worth pointing out that in Jer 31 the new covenant is still with Israel. So it would be misleading to say that the old covenant passed away and there was a new one that included Gentiles. Paul deals with question of the inclusion of Gentiles. His approach is that we were grafted into Israel, not that the covenant with Israel was past.
Yes, the Old Covenant remains, but it cannot be kept: no priesthood.
Yes, Paul views the Christians as being grafted on - which is ok as long as one stops at a mystical level.
Once one starts pretending that Gentile Christians are sinners because they violate terms of the Old Covenant (which were not sins to them before they were grafted in), but that the reason they are not all damned for these new sins is because Jesus forgives them - that's not right either.
Nobody can follow the Old Covenant: a priesthood in the Aaronic line is required, and they were all wiped out with the Temple. And the Old Covenant can't be changed, ever, to allow for the making of a new priesthood. So, it's in force, exactly as written, and therefore a dead letter, at God's own hand, because he removed one of the necessary things to carry it out. All that it ever promised was obedient Hebrews a secure farm in Israel. So in the present state of affairs, nobody can get the reward of the Old Covenant - nobody can get a farm in Israel, secured by God - because the Old Covenant cannot be kept (no priests). The best one can do is buy a farm in Israel, which is secured by the Israeli Army, not God.
The Old Covenant promised the Hebrews (later the Jews) a country and place in the country. If offered nothing in terms of life after death.
THAT is the other problem with the idea that the Old Covenant applies to Christians through the grafting. Not only can't we get the farm, but the Old Covenant never promises hell for eating bacon, or for homosexual sodomy or murder, for that matter. It promises the Hebrews that they will be executed by men, if caught in the latter two of those things (and cast out for the first). And it promises them no farm in Israel. But it says nothing about being thrown into Hell. Hell, Gehenna, the Lake of Fire: those are not revealed in the Mosaic Covenant, and they are not threatened as punishment for sin.
It is Jesus who reveals those things directly. The Hebrews had their legends and myths and their inspirations about Gehenna and the like, but there is not one word about any of that revealed in the Old Covenant. If you broke all 19 Commandments, men might put you to death, and you would not get your farm in Israel, but God never threatened Hell, or even revealed it at all.
The New Covenant reveals a new promised land: Paradise and the City of God after the resurrection, to any who will follow Jesus - which includes not doing certain things he said not to do. The Old Covenant only promises Hebrews in Israel a farm in Israel.
These are strikingly different things. Christians try far too hard to bootstrap the Old Covenant into the New, and to put Christians under fusty rules that Jesus never imposed on anybody.
And Paul? When he was alive, it was still possible to keep the Old Covenant. The Temple was still up, the priesthood was still intact, the sacrifices were still being made. It was perfectly logical for him to think that God would go on protecting his people. Remember, Paul never saw Jesus in the flesh, never heard him speak, and had no New Testament: Paul's letters are the oldest part of the New Testament. So, WE have Jesus in the Temple area, telling everybody that it would all be pulled down and destroyed in that generation, but Paul did not have that history, or any of the Gospel other than what he got by word of mouth. He was mostly running on the inspiration of the Holy Spirit in what he taught, and did not have the detailed history of Jesus that we have.
WE know that Jesus' condemnation of the Temple to utter destruction would indeed be carried out that generation. Paul didn't know any of that, and probably did not even know the specifics of the condemnation. Paul was a Jewish Pharisee, writing to converts in a Christianity that was still heavily Jewish at root, and that still had a Temple, priests, sacrifices. Paul gives no inkling of realizing that all of that will be swept away - that the Temple and the Priesthood and the Jewish religion of the Bible WOULD have its visitation and be removed from the earth very soon, but that this would not herald the end of the world at all. It would, rather, remove Temple Judaism as a viable competitor for attention with the Christianity of the Apostles.
In Paul's day, there was a choice for being absolved of sin for a Jewish Christian: prayer to Jesus, or sacrificing an animal on the altar. Paul did not foresee that the Judaism of Moses would be destroyed forever.
The Judaism of the synagogue, which has gone on after the Temple, is nothing like the Judaism of the Temple. There are no sacrifices for atonement. There are no priests as prophetic lawgivers. There are no Sadduccees and Essenes. All there are, are Pharisees, and all they can do is read a text, all of the rites and rituals and ability to perform them having been wiped out.
Paul never faced that, and he never contemplated it. Paul, a Pharisee, had to try to reconcile and rectify his own Pharisaic Judaism with the Jewish Messiah, who was very different than expected. He never conceived that God would actually ANNHILATE Mosaic Judaism by erasing the Temple and the priesthood completely.
The Mosaic Covenant, exactly as written, goes on forever, unchangeable. Which MEANS that when God sent the Roman Army to blot out the Temple and exterminate the Aaronic line of priests, God removed Templar Judaism from the world forever. There is no competitor left to Jesus' New Covenant. The Old Covenant is intact, but it is physically impossible to keep it. There is no Temple. There are no priests. And the Covenant can't be changed to make a new way. The old way has to be followed, and God killed all the priests so it CAN'T be. Which means that the Old Covenant is alive and intact, but the religion it structures has been dead since 69 AD and cannot be resurrected unless an Aaronic priest can be resurrected. Only God can do that, and he didn't.