Well, I am not sure how you can say it was never about quantity if God desires all to be saved. He wants everyone to be in heaven. That's a mainstream Christian teaching. But, for whatever reason, all God gets is the remnant. How is that not a failure, considering he wants all saved?
God wants everyone to be in heaven? Really? Where does it say this specifically? If you're thinking of 2 Peter 3:9, you'll have to wade through a conceptual swamp of 3 whole chapters to pull this one verse out and understand it within the overall biblical context in which it is situated.
(THEN, you'll have to hermeneutically pull the whole letter of 2 Peter out from the historical context in which it may have been written and haggle over it's authenticity with the rest of us, especially since it is, and has been, a 'disputed' letter.)
No, I think it's safe to say that what it "means" in 2 Peter 3:9, and the similar allusions about the whole world being loved by God which we find in
the Gospel of John, is that God is willing to take those persons--whomever they they may be--whom He has called to come to Him for salvation. He's patient and He waits for us. But it doesn't mean that God will directly chase after every single Tom, Richard and Harry in this lifetime in some obviously direct and physical or verbal way.
Of course, being the kind of Inclusivist that I am, I'm only able to say that those persons (probably mostly men

) who directly and knowingly refuse and reject Jesus Christ straight out are in danger of eventually landing themselves in Hot Water. The rest of humanity who truly has "never heard of Christ" will be under another form of Judgment; it'll be one that, in the here and now, I don't know much about and since I can't have any complete insight about it, I can't say for sure whether it will bode well or not for these lesser enlightened individuals. They will remain a
Big ? in the understanding of the rest of us until all is said and done by Christ and God the Father.
But again, the rejectors of Christ....................well, they will have had their opportunity and they will have squandered it, and their failure in this regard, hand in hand as it will be with the Devil's work, will be ultimately on them.
It's not a failure on God's part if much of the world of humanity willingly chooses to turn its backs on Him. No, it's not His fault. It might be partially the Devil's fault, and it might partially be any one human being's fault, but it's not God's fault.