Modern temperature records are not the only records. One can draw conclusions regarding climate by examining things like historical accounts, tree rings, varves, ice cores and the fossil record. So, yeah, there are records of some sort or another going back as far as the earth goes back. Granted, the farther back you go, the less exact our conclusions will be, but that is OK, as we are understandably most interested in how climate has and will effect human populations. For instance, knowing that the earth was warm enough to form a worldwide magma ocean during the iron catastrophe is not of particular interest in our discussions of temperature changes in the modern earth's climate.