If you seriously deal with it, i.e. repent and spend time seeking him about the issue, then there's no reason to worry. We are in training for righteousness here on earth, and God does know that we're fallible. But at the same time, it is a serious matter. God will never break a promise to us, and He expects the same in return. Sometimes there may be a consequence as a form of discipline, or perhaps He'll show us mercy, it all depends. But, He'll always respond as a loving Father, and He always knows the best way to handle it.
A long time ago when I was a newly born-again Christian, I broke a vow to God. I promised I wouldn't do something, and felt God get my attention immediately, showing me just how seriously He took it. I had confidence in myself and reassured Him I wouldn't break it. But, I ended up doing so anyway not long after. I felt completely devastated afterwards, but confessed it and spent a lot of time in deep prayer, and God did restore me. It took awhile to fully restore my relationship with Him, but over time it did happen. It had enough of an impact on me that I've never made a promise to Him since, and am more sober minded when it comes to making a vow like that.
I do believe He allowed it though to show me not to have any confidence in myself. I learned a valuable lesson from that experience, in more ways than one, and learned that true change comes from yielding to God and renewing my mind with His word. God is the one who changes us.