The Lord is speaking of the Jews and Gentiles. The branches that are cut off are the Jews who still abide in unbelief. The Gentiles, us, are the grafted into the vine. A parallel passage is Rom. 11:14-17
(Rom 11:13) For I speak to you Gentiles, inasmuch as I am the apostle of the Gentiles, I magnify mine office:
(Rom 11:14) If by any means I may provoke to emulation them which are my flesh, and might save some of them.
(Rom 11:15) For if the casting away of them be the reconciling of the world, what shall the receiving of them be, but life from the dead?
(Rom 11:16) For if the firstfruit be holy, the lump is also holy: and if the root be holy, so are the branches.
(Rom 11:17) And if some of the branches be broken off, and thou, being a wild olive tree, wert graffed in among them, and with them partakest of the root and fatness of the olive tree;
The Lord Jesus always dealt with religious things never physical things. He dealt with eternity not temporary. We do the same when we preach the Gospel.
The ones broken off are the Jews as a people. They cling to their false religion but some will be saved and grafted in the same as us Gentiles.