The reference is to Dan.1:3. The word could be translated "eunuchs," but here it is in reference to the royal officials of Nebuchadnezzar before Dan, Shad, Mesh, and Abed entered the Kings service. Even if you could argue that "eunich" was in reference to Dan, Shad, Mesh, and Abed, verse 4 pretty much cancels out the possibility of castration, since it says that they had no physical "defects."
Incidentally, v3 could also indicate that these men were all of royal descent.