Location is the biggest problem, which is unfortunate since the new stadium (if it gets built) looks like it will be in the same place as the Trop. Truth be told, the Trop is actually a pretty nice stadium. The ring supports are a problem, clearly, but as for game day experience and the actual fan design of the stadium, it is nice. I like it quite a lot and don't have any issues with it. However, St. Pete is across the bay from Hillsborough County, which is the main population center, and across the bay from the toll bridge to Bradenton and Sarasota, which are also large population centers.
These bridges can have epic traffic most days from around 3:30-7pm, people just don't want to deal with that. Even in light traffic it is still a 30-40 minute drive to St. Pete from Tampa. There is a water shuttle in the spring and fall, but it is slow as the state mandates the speed on the bay, so it can take an hour to cross the bay via water (even though the vessels are capable of much faster speeds). Add time to park and wait for the shuttle bus and you are at 2 hours travel from Tampa, easily.
I am a Rays fan, I watch most of the games on TV, but I only go to a handful of games a season. The tickets are cheap, as low as $10 in the lower bowl, but the concessions are expensive. But the drive time is the biggest factor. I just don't feel like driving over there after work and getting back home late after the game.