Physical harm is not the only kind of harm.
Eternal salvation is living in union with God who is love. In order to be united to Him in love, we must receive His love in faith (trusting that it is truly what is best for us) and abide in it.
When we do things contrary to authentic love it is essentially a choice to live contrary to God's love--it is a rejection of it. This is what sin is. In fact, many saints have described sin as loving the wrong way--such as loving ourselves, our reputations, pleasure, or material possessions, more than God and neighbor. It is harmful to our well-being as it wounds us in our very nature, as creatures of God who bear His image. This is the case whether we do physical harm to someone else or whether we commit the sin in the secrets of our own hearts. It also puts us at enmity with what is actually out Supreme Good.
Homosexual activity in particular is contrary to our very nature as human beings created male and female in the image of God. It is a perversion and abuse of something set aside by God for a sacred purpose--much like fornication, masturbation, lust, adultery, and other sexual sins are. God's love, especially as expressed in the relationship between the Word and the Church, is reflected in conjugal love between a husband and wife like a mirror. That act bears the image of God's eucharistic love for His people--the giving of Self with the potential to bring forth new life, supernatural for God, natural for us.
To act contrary to that model is to act contrary to God who is love. This is true even for married couples who abuse their sexuality even within the confines of marriage. To invite one's beloved to engage in behavior contrary to love is not love at all--as true love seeks the greatest good for the beloved regardless of the sacrifice necessary (I think my signature is sums up this thought, in regards to love of God at least, but love of another human being is no different).
We all have crosses we must bear--everyone does, and often the one's that are least visible are the most tragic. But they can be our salvation or our damnation. It is no doubt that those attracted to the same sex have a difficult cross, but we all are called to bear our crosses like Christ did on Good Friday so that we may rise in communion with the love of God like He did on Easter Sunday.