It can wear a man down if his wife gives him constant criticism. The worse danger is that he'll actually believe the criticism if it is unfounded. Even if he is wrong, you can't keep a steady stream of it up. If you point out something he does wrong, do it respectfully,
In I Timothy 5, Paul tells Timothy not to rebuke an older man harshly, but to entreat him as a father. That could mean an elder, as in elder of the church, as well. Even though Timothy was in a position to appoint the bishops/elders in the church in that city, as a younger man, he still had to show respect to his elders. He couldn't rebuke. He had to entreat.
The Bible tells wifes to reverence/respect their husbands. If he's sinning, of course you point that out. You have to reprove the deeds of darkness, but with your husband, you need to entreat, not rebuke harshly. You shouldn't speak to him as you would to a child for example. Try to do it without a lot of negative emotion or finger-pointing. Don't repeat yourself a lot or stretch out the conversation too long. Keep in mind you point things out so that he can hear them, not for you to emotionally satisfy yourself by talking about them.
If a man's wife constantly finds fault and the only topic of conversation she has to talk about is "what's wrong with you", he can easily find good reasons to stay in the office and work on something or find some project that will take him away from his wife's mouth. He may shut down conversations and she may say, "Why don't you listen to me?" If her words are unprofitable and hurtful, and do more harm than good, why would he want to listen to what she has to say. That is why it is important to build him up. If his wife tears him down, it can hurt every area of his life. I worked for a while as an electrician's helper when I was younger. One of the helpers or apprentices worked with an electrician who found fault with everything he did. Maybe the kid was doing everything wrong anyway, but it seemed like the criticism made his work worse. It was like he felt like he couldn't do anything right, and it was a self-fulling prophecy. If a wife is always criticizing her husband, especially if he believes her, it can effect the quality of his work on the job, which will eventually effect his wife and the whole family, since it effects the finances of the whole family.
A man in this situation may look back fondly to the days of dating or early in marriage. Back then, his wife actually liked him. I have a relative whose wife can be a bit of a shrew at times. She was in a mood, and everyone was gathering around for a photo. He said to her right before the picture, "Pretend like you like me." That was pretty pitiful, but if a man's wife is always putting him down, he may wonder what happened to the kind, loving woman he married.
If a man's wife speaks kindly to him, praises him when he does something good, uses humility and tact when pointing out his flaws, and seeks to please him as her husband, then when he comes home, instead of finding his wife to be an emotional drain, she recharges him. Husbands and wives need to find ways to recharge each other.
I agree with what a lot of posters have said. Make an effort to think of good things you can say about your husband. You can think of those areas you mentioned where he has improved. Set yourself up for some accountability, too. This may require you to humble yourself, but have a serious talk with when where you admit to him that you have had a problem of saying hurtful things to him that tore him down, and how you want to change. Get input from him on the kinds of things you have been saying that have hurt him or that are just not right and appropriate. Then ask him to point it out if you start doing that again. When he does, try to react well, thank him, and stop doing it.
Cutting people down starts in the mind. The Bible talks about bringing every thought captive to the obedience of Christ. It can be easy to sit around thinking of flaws with your partner, but sometimes criticism aren't really justified. You shouldn't blame every bad thing that happens on your spouse. You may be tempted to assume he had motives he didn't have it things went wrong. Train yourself to be 'fair' in the way you think about your spouse. It also helps to spend some time giving thanks to God for good things about him.
Think about the children of Israel. God gave them manna, and they got tired of it and grumbled against the Lord. Now, this miraculous bread they didn't have to go out and plant. They just went out and it was there, and they complained about it. The Lord judged Israel over that.
We think how could Israel be like that, but if we complain about the blessings God gives us, are we any different? Isn't your husband a blessing from the Lord? It is good to be thankful for our spouse, and one way to helps us cultivate a thankful attitude is to speak out words of thanksgiving in our prayers to the Lord, and thank God for our spouse, and the good things about them.