Did I miss something or was he punished for those specific acts?
Every aspect of Samson's story is pretty amazing. He seems to be the prime example of someone who has the power of God but is short on the character side. He was blessed by the Lord, and the Spirit of God came on him often and in powerful ways. But at virtually every step of the way Samson does things his way, not God's. And God even seems to be in it, using his love interests to bring about defeat of the Philistines.
It was a brutal time, when "every man did what was right in his own eyes". Samson was the last of the judges of Israel, and after him things descended to the tale of the concubine being murdered, with resulting civil war and the tribe of Benjamin being nearly destroyed. There is very little good to say about what was going on back then.
AISI, the whole thing is an exercise in the Lord using very imperfect people to achieve His ends, for their own sake and despite themselves. The only way Israel was going to survive was with a king, though the Lord did not want it that way, because though it would work for a while, He knew how bad it would end up.
And Samson is an example of God's longsuffering as well. He repeatedly violates not only his Nazarite vows, but basic commandments of separation also, such as not going in to non-Israelite women. But God continues to bless him with supernatural power and success, until the day Samson goes too far, and lets his hair get cut off. Then there is a price to pay.
In Romans 11, as Paul concludes his discourse on the Jewish nation, he states that the gifts and calling of God are without repentance. When God gives, He does not take back. He lets things play out in their own way, and uses the process to achieve His ends. (This indeed is true going all the way back to Adam.) We see this play out in our day, when you can have a preacher who is very successful, yet is sleeping with the secretary. God appears to do nothing about the problem, and doesn't pull His Spirit from the ministry. He gets away with it for a while, but the day comes when it all falls apart.
That's what I see happening to Samson. This is the danger of focusing on the external gifts and calling, without tending to the garden of our hearts.
We have a strong warning about this at the end of the Sermon on the Mount,
“Not everyone who says to me, ‘Lord, Lord,’ will enter the kingdom of heaven, but the one who does the will of my Father who is in heaven.
On that day many will say to me, ‘Lord, Lord, did we not prophesy in your name, and cast out demons in your name, and do many mighty works in your name?’
And then will I declare to them, ‘I never knew you; depart from me, you workers of lawlessness.’ -Matt 7:21-23
Through Israel's rise and fall, the Messiah was brought forth at the right time in world history. God using imperfect people to achieve His ends is the theme of the history of the nation of Israel. And indeed, looking around, it is impossible not to come to the same conclusion regarding the church. It's all about God's faithfulness, not our inherent worthiness.
Samson is included in the Hall of Faith in Hebrews 11, so he must have been doing a lot of things right. We have to be mindful as we read his story that God was dealing with people primarily on the national level back then, and in this case was using Samson, warts and all, to achieve His purposes on that level.