What I don't get is why God labels it as immorality. If no one is being hurt why is God against it?
That is an excellent question.
I think the best way to help you fully understand, is to use God's own example.
You know God is faithful, loyal. He would never be unfaithful or disloyal. He remains committed, and will not break a vow.
When God made a covenant - agreement with his people, it became like a marriage vow, so that when they - not God - broke the covenant, they were like an unfaithful wife - an adulteress. See Isaiah 54:5, 6;
As you know, the human family was to be as God is. So, a man and woman were to be committed to each other as husband and wife. That bond of union - marriage - was never to be broken. See Mark 10:6-9
A vow, of commitment in marriage then, was to be permanent. (sin interrupted God's original purpose - only temporarily.)
God's original purpose involved family.
You know this well. God told Adam and Eve to multiply and fill the earth. Obviously not like animals do - with the male threading any female he finds

Rather, the children of Adam and Eve would not have ten dads, or twenty moms. They would know Daddy, and Mommy. What is more, that family would be together permanently. No divorce. See Malachi 2:16
Cohabiting, - as we know - does not work that way. There is no commitment. Anyone can pack up and leave any time, leaving children with one parent, and later, maybe another dad, or mom. Without commitment unfaithfulness does not exist.
The husband and wife provided a balance in the family. See Genesis 1:13
So, fornication - which is not restricted to sex outside of marriage, adultery, and other sexual sins is called sexual immorality, since these things are against God's moral standards.
I hope I did not complicate things, and that was helpful.
