I disagree. There is far too much nuance for a black and white rule.
There are many sports where the physical difference between men and women don't matter (e.g., cornhole). There are many sports where such differences matter but to a small degree (e.g., bowling). Then, there are many sports where they matter significantly (e.g., weightlifting, short distance running).
There are also contexts in which the physical differences are important at elite levels but unimportant at lower levels (e.g., fencing). There are contexts where a male competing against females does not take anything away from the females.
And the more you dig, the more nuance there is.
So to reduce it to a black and white rule is not sensible nor helpful.
Take someone like
Laurel Hubbard. I agree. It's unfair. Hubbard has a significant advantage.
Take someone like
Austin Killips. I agree. It's unfair. Killips has a significant advantage.
Take someone like
Redmond Sullivan. I disagree. It's fair. Sullivan did not have a significant advantage.
Take someone like the athlete in this case. I disagree. It's fair. This athlete did not take anything away from the girls. The girls still received their rightfully earned medals.
I would encourage you to reflect more closely on this statement.