Well, selective recall, I suppose is fine. But I prefer to look at the actual evidence, in this case, the text.
Gen 1:
11 And God said, Let the earth bring forth grass, the herb yielding seed, and the fruit tree yielding fruit after his kind, whose seed is in itself, upon the earth: and it was so.
12 And the earth brought forth grass, and herb yielding seed after his kind, and the tree yielding fruit, whose seed was in itself, after his kind: and God saw that it was good.
13 And the evening and the morning were the
third day.
I do not see where it says every patch of planet earth there. I take it to refer to Eden, primarily, because, on this third day, that is where He planted the plants, far as I can tell. (Excepting some, it seems likely on the earth at large as well, that needed to be there, like some creatures, helping to get it more hospitable)