I would have to say that this is the absolute worst thing you could do. If you want him to respect what you think is important, than you should show him that you respect what he thinks is important. We can't expect what we don't give.
That said, if you want to be late for hangout time because you're in the middle of doing something and you'd like to finish it up first, then you should feel free to do that. It took me a lot of practice to do, but I can do it now. We'll say, for example, that we're going to hangout around 4 on Friday. But I'll end up on a roll with my lesson plans for next week and, rather than stop, I'll keep on going and let him know I'm running late and just say I'll get there when I get there.
I'll admit I've only done that a few times, but knowing that he doesn't mind is actually rather freeing.
And, like I said before, there's flexible times and set times. Limit your stress to the set times and I'll be willing to bet that the problem will largely go away.