That can be a supporting argument. According to historical records it appears that only wealthy men and royalty had multiple wives. It looks as though the general population did not practice polygamy, with the means to support large families being the determining factor.
Another point about ministers being told to only have one wife, the ability to manage a family well is a requirement. Logically, the larger the household the more difficult to manage it would be. There is also the recommendation from Paul (but not a commandment from God) to be single so you can devote more service to the church. So, it looks as though that telling them they can only have one wife is connected to the ability to be valuable and effective as a minister because of a simpler homelife.
One thing to consider is that God requires a lot from husbands for their wives and children. He must be able to meet those need regardless of any other obligations, including the church. If a man is not able to meet the needs of his family, but is giving of himself for the church, he is in error. I personally know a pastor who lost his wife and marriage for this very reason. His church rejected him for it instead of condemning her. He lost everything, his family and his church.