Though exercising a lot (like on the elite level) can mitigate some negative impacts of sugar in our blood stream and in our cells, I am not sure if your statement that, if you exercise at lot, then its not bad and the negative effects are nullified, is completely true.
It seems that carbohydrates act like a toxicity in our body. Using it quickly for energy may lower its negative impact, but the question is to what level, in what situation, after which meals, and in which individual person. I guess it will be pretty much all over the place because the possible life combinations, like age, gender, individual health, type and timing of exercise, the season of the year, genetics/ancestry, various possible carbohydrate forms and meal combinations, are almost unlimited.
Therefore, the safest way for the general, more sedentary population (like me), seems to be to simply not eat carbohydrates or in a very limited amount.