Well you have good evidence that suggests it happened. I don't know at what point a hypothesis becomes a theory becomes a fact. Couldn't you say the same thing about the evolution of life: there's good evidence it happened, but we weren't there to see it, so we simply don't know?
This theory/fact confusion is a very common misunderstanding in science. A theory, in science, is more important than a fact. Facts are observations about reality. They are a dime a dozen. What theories seek to do is explain those facts. They explain why things happen and develop a mechanism which can be used to predict other facts.
Evolution is both fact and theory. There is an observable fact that variations occur at the genetic level. The theory provides a mechanism which explains the diversity of life we find on this planet.
Lastly, just because we are not physically there to observe a phenomena, does not mean we cannot understand what has/is taking place. In actuality, evolution is a poor example of this because we have observed many instances of speciation (sorry, cannot post link, just google "observed instances of speciation" and there are relevant links gallore). We cannot physically go to the sun and see that the cores of stars are gigantic nuclear fusion reactors, turning millions of tons of hydrogen into helium every second, but we can use the evidence we have available to us to discover and explain these natural phenomena.
Now to add something: The reason why we are, as of yet, unsure as to what happened t=0, if it ever did, is due to the fact that, before a certain point (10^-43 seconds after the initial expansion of the universe), the four fundamental forces were one, and to hypothesize on what would occur under such situations would require a unified theory (ie. quantum theory of gravity).
Observationally, we can only "see" back to the microwave background radiation of the universe, approximately 380 thousand years after the "big bang" because at this point, the universe was opaque. However, this observation was made after the initial development of the big bang theory, and the theory predicted the existence of the background radiation itself... very

stuff.