Don't challenge me, just ask.
In his Comprehensive History of the Church (CHC), LDS historian and Seventy Brigham H. Roberts quotes Martin Harris, one of the three witnesses whose name is found in every edition of the Book of Mormon since its original edition. Harris said that the seer stone Smith possessed was a "chocolate-colored, somewhat egg-shaped stone which the Prophet found while digging a well in company with his brother Hyrum." Harris went on to say it was by using this stone that "Joseph was able to translate the characters engraven on the plates" (CHC 1:129).
Martin Harris was one of the scribes Joseph Smith used to record the writing on the plates. This enabled him to give a first-hand account of how Smith performed this translation. Harris noted, "By aid of the Seer Stone, sentences would appear and were read by the Prophet and written by Martin, and when finished he would say 'written;' and if correctly written, the sentence would disappear and another appear in its place; but if not written correctly it remained until corrected, so that the translation was just as it was engraven on the plates, precisely in the language then used" (CHC 1:29). Harris' description concurs with that of David Whitmer, another one of the three witnesses whose testimony appears at the front of the Book of Mormon. Whitmer details exactly how the stone produced the English interpretation. On page 12 of his book An Address to All Believers in Christ, Whitmer wrote, "I will now give you a description of the manner in which the Book of Mormon was translated. Joseph Smith would put the seer stone into a hat, and put his face in the hat, drawing it closely around his face to exclude the light; and in the darkness the spiritual light would shine. A piece of something resembling parchment would appear, and under it was the interpretation in English. Brother Joseph would read off the English to Oliver Cowdery, who was his principal scribe, and when it was written down and repeated to brother Joseph to see if it was correct, then it would disappear, and another character with the interpretation would appear. Thus the Book of Mormon was translated by the gift and power of God, and not by any power of man."
In volume two of "A New Witness for Christ in America," LDS writerFrancis Kirkham notes that Joseph Smith's brother Williamalso confirmed the use of the hat and seer stone. His account is also similar to the accounts given by Harris and Whitmer although he refers to the seer stone as the "Urim and Thummim." He stated, "The manner in which this was done was by looking into the Urim and Thummim, which was placed in a hat to exclude the light, (the plates lying near by covered up), and reading off the translation, which appeared in the stone by the power of God" (2:417). William's account leads us to wonder why Smith went through the bother of digging up the alleged plates if he didn't even have to look at them during the "translation."
Martin Harris, David Whitmer, and William Smith all agree that Joseph Smith used a seer stone and a hat in the translation process.