I think it's safe to say that no one has ever "taken the Bible literally" as though every single thing is supposed to be read in a literal way. This becomes evident when even the most ardent "literalist" still recognizes that there are figures of speech. Nobody, for example, reads the Proverbs and comes away concluding that God's name is literally "a strong tower", or that Jesus is literally a juvenile four legged woolly ungulate.
I think what is often intended by the word "literal" isn't literal, but rather that one should take the Bible seriously. But depending on our respective tradition/denomination and theological persuasions we believe that understanding parts of the Bible a particular way is taking it more seriously than views contrary to our own. So a Young Earth Creationist might conclude that a Christian who reads the first chapter of Genesis as non-literal isn't taking the Bible seriously enough and is trying to compromise their trust in Scripture; though that isn't really the case.
For the most part, though our interpretations may different radically, we all are genuinely trying to take the Bible seriously as the divinely inspired written word of God. And that not taking the Bible seriously really doesn't come down to blanket statements like "the Bible is literal", but rather about affirming the central truths of the Christian faith: Jesus Christ, the only-begotten Son of God, born of the Virgin Mary, suffered under Pontius Pilate, crucified, buried, dead, risen on the third day, ascended into heaven, seated at the right hand of the Father, coming again in glory to judge the quick and the dead, whose kingdom is everlasting.
"If Christ is risen, nothing else matters. And if Christ is not risen—nothing else matters." - Jaroslav Pelikan
-CryptoLuthean