Of course it sounds silly, which is my point, how silly it would be if Paul were to use a fictional character that didn't exist to explain that was how sin came into the world.
But death reigned from Adam to Moses, even over those who did not sin after the pattern of the trespass of Adam, who is the type of the one who was to come. (Romans 5:14)
I don't disagree that we are all sinful, but the point is apostle Paul was trying to explain how death came through Adam and how life comes through Christ:
For since death came through a human being, the resurrection of the dead came also through a human being. For just as in Adam all die, so too in Christ shall all be brought to life (1 Corinthians 15:21-22)
First of all, I would have no problem taking Genesis as allegory as long as the rest of scripture agreed with it. Unfortunately, the rest of the scripture all say otherwise.
Secondly, I don't consider myself a "literalist" and I doubt if there exists anyone who would interpret the whole Bible literally either. I decide whether a passage is allegory or not by looking at the context and cross referencing other passages, to decide whether it is a poetry, parable, metaphor, prophecy, biography, testimony, historical event, etc. But treating the whole book of Genesis as allegory simply doesn't pass the test, even Jesus referred to them as historical events:
in order that this generation might be charged with the blood of all the prophets shed since the foundation of the world, from the blood of Abel to the blood of Zechariah who died between the altar and the temple building. Yes, I tell you, this generation will be charged with their blood! (Luke 11:50-51)
But from the beginning of creation, ‘God made them male and female. For this reason a man shall leave his father and mother [and be joined to his wife], and the two shall become one flesh.’ So they are no longer two but one flesh. Therefore what God has joined together, no human being must separate. (Mark 10:6-9)