The "gifted" program at my school consisted of coloring book sheets. Mind you, I was in eighth grade. I didn't want to spend 45 minutes of every school day using crayons to make pretty pictures.
That...sounds like a Special Ed class, not Gifted. At least not from my experience. But teacher competency is probably a factor.
Our district has/had strong Honors, Magnet, and Gifted programs, so they all fed into each other. Even when I was in elementary school, the Gifted program was more topical and by the end of the year consisted of the students actually doing research projects that pertained to a theme. That Gifted programs engage in alternative learning strategies is obvious on its face, but it still put average/above-average responsibilities for learning on the student.
With the one I attended in elementary, it was actually an out-day, where once a week, the students would be picked up from school and taken to the Gifted center, have a day's worth of activities around the designated yearly theme, and then go back to the school at the end of the day. The Gifted elective I took in 8th grade (which I took only because I didn't want to be forced to take gym) was nothing like the one from elementary, and bore more resemblance to what I'd assume is a Sociology or a general Humanities class, now that I think about it. It was also part of the larger Magnet program (IMAST) I was in, so that naturally played into it too.
There was also one year - I think it may have been 1st or 2nd grade - where the school put me in a Speech class because my pronunciation sounded 'dutchy' (to use my family's wording). From what I can remember of it,
that bore more resemblance to what you described, as it shared the Special Education classroom and the groupings weren't all that cohesive. But after a couple months I didn't have to go anymore - considering my age at the time, I don't know if it was because I grew out of it or what. Considering that I was also put into advanced groups at my regular elementary school throughout this time, I doubt it had anything to do with them thinking any problems with speech = learning disability.