Also, you are trying to tell me that in any frame of reference, all of the other galaxies out there will look like they are moving away from us? Even if we were to look from another galaxy's point of view?
Yes.
If you want to see how this works, there is a 2-dimensional analog to this.
Take an uninflated ballon, and put some little stickers on it. Make them small stickers, as you'll want to put several on it. Distribute them all around the balloon. These stickers are the "galaxies" on the 2-dimensional "universe" defined by the surface of the balloon.
Now, begin inflating the balloon and notice that the space in between all of the stickers is increasing. Imagine now that you are a 2-dimensional inhabitant of one of the sticker "galaxies", and you're looking out at all the other sticker galaxies. What would you observe? That all the other stickers are moving away from you.
Turn the balloon around, and now pretend you live in a different galaxy, and ask yourself the same question. Are all the stickers moving away from you? Yes. Choose any sticker anywhere on the balloon - and you'll come to the same conclusion.
Keep inflating the balloon and see how the stickers keep separating farther and farther apart. And note that no matter which sticker you pretend to be living in, you always make the same observation - all the other stickers look like they're moving away from you.
Now, it should be pretty obvious that the motion of the stickers away from one another is a result of the
space in between stickers expanding - that is, you're literally adding space between each of the sticker galaxies.
Now, ask yourself this question - where is the center of this expansion? If you're the inhabitant of one of the 2-D sticker galaxies, you might incorrectly come to the conclusion, "I'm at the center, because everything radiates away from me." But you'll notice that's clearly wrong - because you could have said the same from any other sticker galaxy, not just one in particular.
The actual center of expansion is
not anywhere on the surface of the balloon - the center, you can probably already guess, is actually in the center of the balloon, somewhere in the interior where all your air is. But, that is a third dimension - the surface of the balloon comprises only 2 dimensions, but the interior plus the surface comprise 3 dimensions.
Thus - this is a
two dimensional universe expanding in three dimensions.
How is this analogous? Although it is nearly impossible for our minds to actually envision this physically,
we live in a three dimensional universe expanding in four dimensions.
This "fourth dimension" is not the "time" dimension as envisioned by Einstein, but a fourth spatial dimension that we, as 3 dimensional creatures, are incapable of observing physically.
And that's how our own universe expands. Space, literally, is increasing between galaxies. It is also expanding between stars, between planets, between individual atoms, etc.
However, the rate of this expansion on less than a galactic scale is so small as to be nearly insignificant on that scale. Even between nearby galaxies it is fairly small, and the individual motions of galaxies (by gravitational attraction) may be greater than the spatial expansion. For example, the Andromeda galaxy, our nearest full-size galaxy neighbor, is actually observed as moving toward our galaxy. The space between these two galaxies is small enough, and the relative motion of the galaxies toward each other great enough, that we observe this.