For myself, it is enough to show such people where these things they are decrying actually come from and how they came to be as they are, and if they still insist that whatever it is that they are objecting to is 'pagan', then okay. What can you do?
I know that our fathers such as St. Justin Martyr, St. Basil, and others did not have such a dim view of everything 'pagan' anyway, and they were writing at such a time when there were still actual pagans about, not re-imagining so many centuries after the fact what such people would have done or believed, as some Christians are eager to do now. And similarly there are also men like my beloved father and baptismal saint St. Shenouda the Archimandrite, who is remembered chiefly for his disputations and confrontations with said pagans, who for the first four centuries or so of Egyptian Christianity were still a cohesive religious and social force in that country (we know that they survived the closing of their temples under Emperor Theodosius in the 380s because the last dated example of Demotic, which was used by pagans since the 6th century BC, is from 452 AD). In fact, no less dependable a source in this area than the famous Coptic historian Dr. Aziz S. Atiya made the point in one of his lectures in the 1970s that for the first two centuries or so, Egyptian Christianity was characterized by a kind of admixture of paganism with Christianity, as Christians reinterpreted what had been their own fathers' religious and cultural symbols such as the ankh/crux ansata, baptizing them into the Church as they themselves came into it; indeed, I have a feeling that the Egyptian church itself would be the ultimate "not Biblical" church in the eyes of many of the zealots the OP describes (after all, Copts especially are not shy about embracing their identities as the original Egyptians, which necessarily means embracing the thousands of years during which they were not Christians yet), which only makes it funnier when you realize that the earliest fragments of the Bible in Coptic that we have today date to c. 150 AD -- both well within the period discussed by Dr. Atiya as being characterized by an admixture of paganism and Christianity, and several centuries before the Greek received text had been established as such for the people who would (mostly) become the EO, in the fourth century or so.
So isn't that curious...this Church which had the Bible before almost anyone else is also the most outwardly 'pagan' due to its mode of being because they do not pretend to have been sealed in a bubble of religious ignorance or something. Hmm.
I've never really understood the point of being a pagan-hunting Christian anyway. I know that some other religions, such as Islam, like to pretend that before their guy came around was 'the time of ignorance' (what the Arab Muslims call 'jahiliya'), but Christianity is not like that, or at least is not traditionally like that. And knowing more of where we came from can only help us to better appreciate how we came to be the way we are, right? This way we can see what has been inherited, e.g., from Jewish temple practices, vs. what is derived/modified from the cultural practices of the pre-Christian populations of a particular place, etc. And there's no shame in any of that, since everyone comes from somewhere, and after all we are more interested in where we will end up. It just seems to me like the pagan-hunting types are only a hop, skip, and jump away from seeing Christianity itself as a re-tread or repackaging of paganism, given all the things that they object to in the name of 'purifying' the religion from the reality of its own history, in which case why bother being a Christian at all? They do not trust any church to have made the correct calls regarding what is baptized into church practice and what is not, and hence they cannot ever really be secure that things are 'pure' enough, so they cut themselves off from the historical life of their own faith and end up, ironically, looking at the oldest communities as the worst off in this regard (since they tend to have more of what this type of Christian unthinkingly writes off as 'pagan', without actually knowing why it's there to begin with, or many times even what it is that they're looking at), preferring instead to follow a Christianity that is rootless and dedicated to nothing more than proving itself 'least pagan', rather than 'most Christian' (i.e., most in keeping with what actual history shows that the early Christians did and believed).
It's bizarre, frankly. The phrase "cutting off one's nose to spite one's face" comes to mind here.
I think as a lurker though, it may have helped (as a third party)...so it isn't completely useless!
