I feel a lot of guilt for this reason, too. I'm a chronic illness person with a sleep disorder, and I tend to have my worst "feel-like-I-got-hit-by-a-truck" pains and exhaustion in the morning. I try to go to Saturday Vigil Masses because that gives me more time to work through my pain and I can usually drive by then, but my work schedule gets in the way of that fairly often.
I think I could make more Sunday morning Masses if they were closer to my house, even with a moderate amount of pain. But it can be too much to deal with the pain and brain fog and exhaustion driving on the interstate for so long where so many people drive recklessly. I don't want to cause an accident or be too tired and slow reacting in defensive driving to get away from one.
Still, no matter how much pain I am in, or how tired, I always second guess myself and feel like I should have tried to go. I've been scrupulous about going to confession for the missed Masses, even though I knew I wasn't feeling well. In response, I have been repeatedly told in confession that Mass is a requirement because it is the source and summit of Christian life, which is absolutely true. And this always makes me feel like I didn't try hard enough, because God deserves my very best. It's just hard to know where to draw the line, how much pain and exhaustion is unsafe, because I exist with it in varying degrees all the time, and sometimes God gives me the strength to get through, and sometimes my body fails.
Lately, though, everything has been so overwhelming that I finally stopped beating myself up over this. Michie, I think you are right. God is just, He knows our hearts and He is not looking for reasons to send us to hell. I think I finally made some peace in my mind about this, right or wrong, by realizing that even if I didn't "try hard enough" with all these problems, the force of my will still wants to go, and any sin would probably be venial, if it was a sin.
I look forward to being able to attend Mass in eternity without a mortal shell hindering me.