Then you're back to the position that you have no authority on which to accept the Bible
You're just contradicting yourself.
Adam walked with God, not simply a disembodied voice.
This makes no sense. On what authority could he conclude that it was not a lying vision from an evil spirit? You yourself said that spirits cannot be trusted, hence the pastor was right that our only option is to "check it out with Scripture."
Scripture is the historical record of what the Holy Spirit has revealed to the faithful, we can trust the Bible because it is the witness of the Holy Spirit, and the Holy Spirit continues to affirrm it.
Are we now back to the voice of the Holy Spirit as authoritative Inward Witness?
What happened to the Bible as the only authority?
You're going around in circles, self-contradictory ones.
Ultimately, no epistemology has been shown to escape Munchaussen's trilemma so I see no reason to be concerned.
You should be concerned since you hold to a Sola Scriptura epistemology that already contradicts the experience of Adam and Even in the garden, not to mention the experience of all OT and NT saints, and contradicts the thesis of John 10:27 - namely that we CAN and SHOULD trust the voice of Christ, regardless of whether Scripture exists or is understood.
And what is the right question?
Here it is - wait for it:
To what extent did the voice or vision convict/convince you? What level of certainty did it bring you to? 100%? If not 100%, did it at least raise you to a degree of a certainty such that you cannot question it in good conscience? If so, you are morally OBLIGATED to accept it as truth.
This is how the voice (the Inward Witness) has always functioned, for Adam and Eve, for all the prophets, for all saints in both testaments. This is the only dynamic that makes sense.
It is on THAT authority - that voice - that I accept the Bible. Meaning, I accept the Bible because the Inward Witness has apparently left me too convicted/convinced about the Bible to repudiate it in good conscience.