Definitely, if you feel that God is calling you to give up fantasy books, you should do so.
On the general issue of the goodness or badness of the fantasy genre, I would submit that there is nothing wrong with fantasy as fantasy, but that it is important to hold it to some fairly high standards as literature. Basically, I consider it essential to consider what constitutes good literature, and evaluate any work on such a standard, weather it's fantasy, Christian fiction, thriller, historical fiction, or whatever. First, my definition of "good" may include style, because I have little patience with clumsy writing when there's so much better stuff out there, but is primarily based upon the piece's ability to portray Truth. A piece of literature has the capacity to show us essential truths about human nature, the nature of the universe, and shine a light on many dark places; sometimes for the good, sometimes not. Every genre presents it's own natural strengths and weaknesses; biographies may present startling pictures of what Man is, in all his highs and lows, fiction may show in a scene, in a story, the wisdom philosophers speak about in all their volumes (Read Dostievsky or Les Miserables). Fantasy, at it's best, presents a unique, wonderful, and all too often misused opportunity to reflect the image of the Creator, through creating ourselves - little microcosms which may either illumine or obscure Truth, Beauty, Goodness, Love, and the darknesses which must be opposed. The thing to consider is, what is this particular piece using it's power for good or ill? Is it an honest attempt to reflect the glory and adventure of creation, or merely a cheap thrill ride? I would say that the main failing of Christian fiction generally (I haven't read much Christian fantasy) is merely that the modern stuff is too safe, too bland, is not willing to take a risk, state plainly the real risks, the real darks and lights with any power or conviction; frankly, they lack heart and style. There are probably some good ones, I just don't know them.
Of course, my theory doesn't always come through in my choices; I make mistakes, and read lousy novels (Most of Terry Brook's books have definitely been a wast of time, as have several of Silverburg's), but, in the end, the reason for reading fantasy os exactly the same as reading any other book, and, ideally, all that we read may be edifying. From my experiences (admittedly not the broadest) I would recommend Phantestes and Lilith by George MacDonald, The Lord of the Rings (of course), The Song of Albion, by Stephen Lawhead, Till We Have Faces by C. S. Lewis and maybe (I'm not quite sure about this one) the Chronicles of Thomas Covenant by Stephen Donaldson - most of the other good books I've read have not been fantasy.
I hope all that made sense... :o