Pamela Geller and Robert Spencer were coming to address an English Defence League rally. Terry Jones was coming to address an England is Ours rally. These were anti-Muslim demonstrations, decrying Islam in all forms and not merely limited to extremists, including existing UK citizens.
I have no problem with excluding non-citizens from entry to the UK who want to come here and publically advocate that UK citizens should be censored or driven out due to their beliefs. If you think that is a "free speech" issue, I'd say freedom of belief is a more fundamental and intrinsic right than that.
Savage was banned at an unspecified point in time (in or before 2009) for "seeking to provoke others to serious criminal acts and fostering hatred which might lead to inter-community violence". You may call that free speech, incitement to criminal acts is a crime here, as indeed it is in your own country in many cases.
Julien Blanc was banned because “To allow this man into the UK legitimises sexual assault and predation, and sends a message that women are playthings or objects without agency.”. To pick one quote that I feel summarizes him quite nicely, he refers to his time in Japan as: “If you’re a white male, you can do what you want. I’m just romping through the streets, just grabbing girls’ heads, just like, head, pfft on the d**k.” If he wasn't a citizen, would he be allowed in the US? Some of his own descriptions of his actions are descriptions of sexual assault, as defined by US law.
Say what you like about free speech in the UK, at least we don't have any apparently popular candidates for government proposing that those of a certain faith face a blanket entry ban due to their faith alone. I'd classify that as a speech issue. It seems to me you don't?