That would be OK except for a couple things:
1. That doesn't deal with the networks between the two points. I might be using an ISP that deals with all traffic equally, but lets say some network I pass through to get to a site is owned by TimeWarner, but I'm on Comcast. Since Timewarner doesn't like Comcast, they decide to block or slow the traffic meaning my access to $SITE is slowed.
2. There's simply not enough choice available to make this a viable option. Around me I have the option of dealing with either Comcast or Verizon for high speed internet. Actually at the moment I only have the option of using Comcast. Other than that, there's satellite which prices most people out of the market, or dial up, which is too slow. It would be more viable if it was like back in the early 90s when there were tons of dial up ISPs and you had a wide choice between similar services.
Though it would be nice if ISPs gave me the option of assigning bandwidth priority on my own line. If I'm connected to multiple sites doing a variety of things, I'd like to have the option to declare one as a higher priority. So that way if I'm, for example, streaming music while reading message boards, I can make the streaming the priority so I don't have the music skip as I navigate around the board. An extra half second while I wait for a static page to load is worth it if I gain seamless music.