Some people, such as myself, love odd-ball, unique voices. I appreciate artists who can't hit all the notes, who don't know how to stay in pitch consistently...but sing from the heart. An off-kilter singer who is stumbling over his words but pouring out his soul is much more impacting to me than some really pretty voice singing a mathematically written pop melody.
Some people are just attracted to the weirdos, the underground, against the grain, different, anything-but-accessible/popular.
All but one of my favorite artists have either strange voices, or strange vocal delivery techniques (Geoff Rickly of Thursday, Daniel Johnston, Joanna Newsom, Blind Willie Johnson, Ian Curtis of Joy Division...). It's appealing because it's unique and because it's refreshing to hear someone who doesn't care what they sound like or whatever people think, they're just doing what they love. And Bob Dylan is just doing what he loves. He's a rebel and will sing about and talk about whatever he wants, however he wants...and change things whenever he wants (like when we went electric and some fans hated it), because he's a drifter and a rebel and he keeps his creativity flowing for the most part without chains.
His music is mostly just a rythymic back-drop for his words (since he has SO MUCH to say, he's used half the words in the dictionary probably lol)...but when he's playing guitar while singing and alternating to the harmonica, the sound can be very pleasing too. And sometimes you can get lost in some of the drum-loops, such as the rata-tat-tat in "Sad Eyed Lady of the Lowlands". And again, there's the occasional vocal melody which is very enjoyable to sing=along to (like "Tangled up in bluuuueeee".