The whole problem is that a term which, at face value, could be supported by all Christians, has been hijacked by a specific group of Christians and taken to mean far more than it means at face value. For example "creationist". Ideally we could all call ourselves "creationists", even TEs believe that in the beginning God created the heavens and the earth, but the term has been taken to mean creation not only in the ontological aspect (that God created) but in terms of a certain methodology of creation (God created in six days, six thousand years ago). So although it would be entirely honest for TEs to call ourselves "creationists" in the brute literal sense of the word, we cannot really call ourselves "creationists" in the sense of identifying with those who specifically label themselves creationists.
The overall TE position with respect to IDism is pretty much the same. If the word were being used for the very first time without any baggage whatsoever TEs should be quite happy to say "yes, God designed all life, and He did it through evolution". Nowadays, even we humans (following in God's footsteps?
) use evolutionary ideas - genetic algorithms - to design things. The problem is that most people who call themselves IDists today are not just subscribing to the term in a general, ontological sense (God designed, it didn't all just fall together) but to a specific methodology (God designed mechanisms for which there are no possible evolutionary pathways of origin). So while TEs wouldn't mind being called IDists in the sense of being opposed to the atheistic implications of atheistic evolution (as you rightly noted), it would be misleading for us to call ourselves IDists if that would imply that we support the ideas of others who call themselves IDists.
Labels, labels. The lesson is that we cannot just put people in boxes; we need to listen and learn before we know where someone is coming from and blast in the direction.
And you are right, IDism does not necessarily invoke Christianity. In fact, it is compatible with practically anything. IDism even works with atheism. Either through panspermia (intelligent lifeforms independently evolved elsewhere, designed us, and then left us on Planet Earth), or through weird sci-fi ideas (humanity at the end of time becomes so powerful that they reach back in time and seed the planet with its first life-forms). The fact that most ID people are Christians is quite telling in terms of the movement's intellectual loyalties.