We've never directly observed a plate moving, and we certainly can't repeat it. But the theory of plate tectonics states, among other things, that the movement of such plates causes earthquakes. Are plate tectonics not science?
What about atomic theory? No one has ever directly observed an atom. Is atomic theory not science?
You say you took chemistry? I'm sure they taught you about reactions. Did you ever directly observe elements reacting? Actually see the atoms reacting? Did you actually see chemical bonds breaking with your own two eyes? No, of course not. But you still call that science, right?
No one's ever been to the center of the Earth, or even more than a few miles below the surface.
Nonetheless, we have things like this that tell us what the Earth is made of right to the core. We know what composition of our planet all the way through, even though we haven't and likely never will travel beneath the upper layers. Is that not scientific?
Would you say that, just because we've never been to the center of the Earth, that this depiction of the Earth's insides is as valid as the one above it? There are people who genuinely believe the Earth is hollow and that it has a star at its center. Since it can't be directly observed, should they be taken seriously?