There are a lot of psychological downsides from homosexuality, ranging from the belief that one is destined for hellfire due to his/her cultures religion, to feeling of isolation from ones family. Additionally there are feelings less caused by society such as the feeling that one is unable to have natural children with the person one loves, and the knowledge that ones sexuality is vastly different from the general populaces and the higher difficulty of finding a partner is of note.
I personally feel the argument 'There are psychological difficulties with sexuality X, so X is bad' is a horrible one, but if you must use it, you can't only selectively chose to use it against sexuality you personally disapprove of while ignoring it on those you don't.