Can I flip this and answer it as a woman?
For me it isn't about "chances, temptations, etc." but about fleeing from the appearance of evil. I would not find it appropriate to be alone with a married man in a casual setting simply because I would not want the appearance of any impropriety there. I know that nothing would happen. I also have total respect for the guy's significant other wouldn't want her to ever have any cause for concern. I will not even compliment a married man, but if I am close to his wife then I will pay the compliment solely to her.
Keep in mind, I am not talking about being alone in say an interview, or professional situation, or anything like that. I am talking strictly casual and private. Then, I wouldn't.