OK, then do I understand correctly that "evolution" only applies to change within a species? Does it not have to start with something before there can be a species in any category?
Evolution is changes in populations of organisms over time. Species is just a label we apply to organisms to categorize them. Living organisms do not actually neatly fit into the categories we make for them, so evolution is never defined by species, genus, etc boundaries. It wouldn't make sense to even define it that way, because evolution isn't limited in how much change can occur, as long as enough generations have passed.
Evolution as a process only acts on existing life. It has nothing to do with the origin of said life. One could consider the first living organism that ever existed to be the first species if they so desire, though the need for the category would be questionable when only one form of life existed.
Abiogenesis is the leading theory on the origin of life itself. It covers molecules to first living cell. Evolution covers the first living cell onward. To be very blunt, the first cell could have magically popped into existence, and that would have absolutely no relevance or impact on evolution as a process.
Horse evolution seems to come up quite often but the chart I was referred to seems to show it starting with Eohippus and ending with Equus caballus is that in error?
Time-wise, no. However, an unfortunate mistake people often make due to these charts is assuming that these fossil organisms are definitive ancestors of modern organisms. Unless we have DNA for the purpose of comparison, this is actually impossible to determine. What fossils are mostly good for is the demonstration of concept, as well as the history of evolutionary developments (such as when the first mammals began to appear on Earth).
Does that mean it started with Eohippus and ended with Equus caballus and there was no evolution before and has been none since?
Not in the slightest, evolution is a continuous process (though, various factors can influence how fast it is, which is why some populations of organisms seem to have changed very little compared to others). In fact, the conditions needed to halt it are currently impossible to produce, since it demands eliminating mutations and changes in the environment entirely.
Furthermore, the evolution of all mammals, vertebrates, and even all eukaryotic organisms has ties with the evolution of the horse prior to the existence of Eohippus. The reason Eohippus is often the "starting point" of charts about horse evolution is due to the abundance of intermediate fossils between Eohippus and modern horses.