I think it is ok to be both sorry and to also feel they should have been better prepared or listened to you better.
From what you described I would not worry too much though.
That is part of being a doctor
You did not waste his time.
His job is to help you, to heal you, not to demand you make...