I'm a male, and I agree with Paradoxum here, so what you're saying seems to be an overgeneralization. I'm definitely not "physically and mentally built for work". I consider work to be a necessary evil. Even when working on something I enjoy, I'd rather do it without the implied threat of bad things happening to me if I don't do well enough and lose my job.
But it's not either work or doing nothing at all. You can do all kinds of stuff.
Read books. Write a book. Play a sport. Volunteer. Learn a new language. Go traveling. Play an instrument, draw, sing. Learn programming and create a video game. Get yourself some Arduino and build electronic devices. Tinker with your car. Make friends. Spend time with family. And literally hundreds of other ways to spend time in a productive way.
There are just so many possible things to do. Work isn't life, it's something that gets in the way of life. So I think the best way for society to progress is to make sure that people have to spend as little time working as possible. I believe that technological advances in automation, as well as experimenting with new political ideas like basic income and a shorter work week, could be a way to achieve that.