There was nothing wrong with Cain's offering except in the mind of the author.
Yet the author is God and he is not wrong so there was a problem with Cain's offering.
Cain's sacrifice was rejected because he lacked the faith to offer the best of what he had. He lacked the faith of knowing that God would take care of him despite his loosing the best of what he had worked for. His offerings were low quality and he kept the best for himself unlike Abel. The answer is Cain had a lack of faith, offered a non-acceptable quality and/or amount of proper offerings.
Gen 4:3 And in process of time it came to pass, that Cain brought of the fruit of the ground an offering unto the LORD.
Gen 4:4 And Abel, he also brought of the firstlings of his flock and of the fat thereof. And the LORD had respect unto Abel and to his offering:
Gen 4:5 But unto Cain
and to his offering he had not respect. And Cain was very wroth, and his countenance fell.
Gen 4:6 And the LORD said unto Cain, Why art thou wroth? and why is thy countenance fallen?
Gen 4:7 If thou doest well, shalt thou not be accepted? and if thou doest not well, sin lieth at the door. And unto thee shall be his desire, and thou shalt rule over him.
Gen. says he didn't "do well" regarding his offering. Add that Hebrews says he lacked faith, the picture becomes clearer especially contrasted to the obviously good quality his brother offered. Abel's offering deserved respect.
Cain's offering did not deserve respect.
Heb 11:4 By faith
Abel offered unto God a more excellent sacrifice than Cain, by which he obtained witness that he was righteous, God testifying of his gifts: and by it he being dead yet speaketh.
"more excellent"
G4119
p?e?´??, p?e????, p?e´??
pleio¯n pleion pleon
pli'-own, pli'-on, pleh'-on
Comparative of G4183;
more in quantity, number, or quality; also (in plural) the major portion: - X above, + exceed, more excellent, further, (very) great (-er), long (-er), (very) many, greater (more) part, + yet but.
Cain's offering was not of the same quality as his brother Abel's was. A lack of faith is the reason why Cain did not give as good of a sacrifice. Keep in mind the meaning of sacrifice. No one is really sacrificing much when you give away the worst but to give away the best that you have and have faith that it will be ok, that is true faith and a true sacrifice.