Psst. Hey buddy. That's supposed to have been the argument from absurdity, i.e. "Your position is so stupid that you can justify literal Potterism". You don't then turn around and say, "Yeah, that's fine, to that person Harry Potter is literally true", you instead pause and try to understand how the heck your position went so far off the rails!
Fundamentally, you've admitted that your logical construct can admit literally any position, regardless of how insane it is. Last Thursdayism, Church Of The Subgenius, Islam, Christianity, Scientology, Potterism, Satanism, none of these can be in any way distinguished by your thought process. It can entertain both an epistemology founded on "X=FALSE" and an epistemology founded on "X=TRUE". It is
utterly useless. You need to rethink your position, and come up with a better way of judging epistemologies than, "Welp, hard solipsism can't be solved yet, so I guess anything goes!"
And if I smacked you in the face with a baseball bat, I'm sure you would find this argument
extremely unconvincing in court. Or am I wrong? If so, allow me to procure said baseball bat, because this is perhaps the one case where Argumentum Ad Baculum is
not a logical fallacy!