I hate to note this since some of the most awsome TE posters here accept a literal Adam and Eve*, but your assertion is problematic since if we go by your assertion, either you're espousing pre-Adamite humans and a type of Gap Theory, or you're denying a literal interpretation of Genesis where already, at the second generation, Cain and Abel were farmers or animal husbanders with nor reason to revert back to the hunter/gatherer lifestyle of "cavemen."
In your dilemma, either there were pre-Adamite hunter/gatheres, or there was a reversion to hunting and gathering that occured some time after Cain and Abel, which you'll need to provide some sort of reasoning/evidence for.
*I've had several interesting discussions with them regarding the Fall as being an advent of agriculture/animal husbandry vs. a pristine primordial hunter/gatherer society so it's not a dig on literal Adam/Eve TEs btw.