As I see it there are several questions that tend to get confused when people talk of literal interpretation.
One is understanding the sense of the text. Is it history, poetry, satire, parable? Rather than literal I prefer the term "plain sense," what a reader at the time would understand. Most of the time I think that's clear. In a few cases there are disagreements. E.g. I think Jonah is an obvious satire. Some think it's historical. But that kind of disagreement is fairly rare.
A second question is historical accuracy. As far as I can tell, Genesis is narrative. I don't think it's symbolic or a metaphor. But that doesn't mean it's historically accurate. At this point I have a slight uncertainty. I think it's likely that the editor of Genesis knew he didn't have the same kind of historical records for that period as for the kings. He may well have understood the early material as legend. Is that "literal?" In some sense yes. It's not metaphor or symbolic. It's straight narrative. But not necessarily history. Of course even where the original author or editor thought it was historical we may disagree.
A third question, which is often the most important, is how it applies today. The OT has lots of laws. It's pretty obvious that they're intended as laws. Again, not metaphor or symbolic. But that doesn't mean they apply to us. There are various reasons that commands given in the Bible, both OT and NT, might not be commands to us. (The case of NT commands is typically more controversial, so I'm not citing specific examples here.)
Then there's questions of detailed exegesis. Did Paul intend to teach election in a Calvinist sense in Romans? That requires looking at historical background, meaning of words, and understanding the whole thrust of Paul's argument. It's not a question of whether the text is literal or not.