What a great question. It also ties into important questions about what is and is not inherently and truly sinful in general.
All sorts of time consuming and often expensive personal interests: sports, concerts (oh no, surely you don't mean baseball or football too, do you?), events, entertainment, etc. Lions and tigers and bears - surely that must all be sinful, yes?
The answer appears to be - if it is not explicitly stated to be sin in the Bible, then it is not intrinsically or inherently evil or sinful.
Here's the part that many people seem to be mistaken about too: the same goes for things which people claim to be sin, but which are not stated as such in the Bible. Things like gambling (yes, that's right, gambling), drinking not to excess or dissipation, etc.
Each person and situation is unique. So while these things may not be inherently sinful, they can be sinful in context for one and not another.
Can you afford to be gambling right now? If not, there appears to be a spiritual problem. Are you doing it out of idolatry or the love of money? Problem. However, is it merely something you can well afford and which is not a problem part of your life? $100 or $1000 may be too much to risk for one person and extremely foolish, but may be almost nothing to a more wealthy person who is supporting a family and giving lots of money still. Okay... Admittedly something like that is not exactly the most noble thing we can be doing with our time, but the same goes for all the other entertainments people hold so dear and often spend far more time and far more money on than those who engage in something like gambling, for instance. There may be no less idolatry and foolishness and may often be far more in things like sports, music, outings, travel, home remodeling, decorating or expansion, etc.
As for James 4:3, it seems to me that looking at the verse rightly, it's about the reasons why we ask for the money, related to what we are living for and loving in our hearts. Are we worshipping as an idol or living for these various pleasures in life? Then it seems we are asking "wickedly," that we may spend it on our pleasures, as I have seen it translated literally according to NAS. Do we have a reasonable and sensible attitude about life's pleasures if we are fortunate to have access to any of them, but is our main focus for asking for the money a good and reasonable one not focused on idolatry of things or pleasures, and can we see that in ourselves by God's grace, and is it clear that He can see what is in our heart? This seems to be the key.