I have always thought this theory was by far the most interesting. I own many books that discuss this possibility. I thought I would share some quotes.
"There is an ancient Hebrew tradition, which of course would not have been affected by the passage in the Book of Hebrews, that Melchizedek was actually the patriarch Shem, still alive during Abrams day. Assuming there are no gaps in the genealogies of Genesis 11, Shem would have lived until thirty-five years after Abrahams death, so that this would be possible. The name Melchizedek would, in this case, be regarded as a title rather than as an actual name. It does not seem unreasonable to imagine that, after the Dispersion at Babel, Shem might have moved, under divine guidance, to the place where God would one day establish His temple. As the custodian of the patriarchal records, he could then have transmitted them to Isaac after Abrahams death. This would also help explain why there was no document entitled the generations of Abraham. Terah, according to Genesis 11:32, continued to live in Haran for sixty years after Abram had left for Canaan, and therefore also was still alive at this time. Isaac was thirty-five years old when Terah died and forty-five when Shem died, again assuming no gaps in the genealogies."
Henry M. Morris, The Genesis Record : A Scientific and Devotional Commentary on the Book of Beginnings (Grand Rapids, MI: Baker Books, 1976), 320.
"Both the second-century Aramaic Targum Neofiti and the Fragment Targum identify Melchizedek with Noahs son Shem in its translation of Genesis 14:18: The king of Righteousness (Melka-sedek), the king of Jerusalemhe is Shem, the great onebrought out bread and wine, for he was the priest who served in the High Priesthood before the Most High God (also the later Targum Pseudo-Jonathan).
Later rabbinic traditions understand the high priesthood to have first been given to Shem-Melchizedek (Gen 14:1820) but then transferred to Aaron through Abraham. Psalm 110:4 was read, You [Abraham] are a priest forever (see b. Zebaḥ. 62a)."
Stanley E. Porter and Craig A. Evans, Dictionary of New Testament Background : A Compendium of Contemporary Biblical Scholarship (Downers Grove, IL: InterVarsity Press, 2000).