I think it all depends... not only upon age itself, but it also depends on the differing opinions among those of roughly the same age as to what would be considered "old". Lifestyle and background could play a big role, too. Sometimes it's flat confusing. It's basically boils down to how we, as individuals, feel. I hope this makes sense.
The OP makes a good point: that a 20-year-old would believe that we in our 40s (and sometimes early 50s) are "old", while a 60 or 70 year old would say "just a baby". I am going on 42, so I am right smack in the middle. Elderly people, upon learning my age, tell me, "Oh, you're just a pup", or something like that. On the flipside, I recently placed an order at a restaurant and found on the receipt where I was automatically given a senior discount (but of course, the kid who took my order didn't appear to be any older than 19)! Also, I remember when I was about 19, and I used to hate it when people told me I looked 12. Throughout my 20s (and even into my early 30s), people continued to guess me as being a teenager. One time, when I was 31, a lady asked me what grade I was in! It really bugged me at the time, as I was clearly an adult and didn't want to be seen as a child. Now, I take it as a compliment when I'm told I look even 10 years younger than I am. It doesn't happen all that often, but when it does, I totally eat it up!
In conclusion, my basic perception of "middle age" is roughly between very late 30s to about 54-55. That's a big range, yes, but it seems as though the older part of one so-called "generation" can tend to "overlap" with the younger end of another (in terms of relating to one another, reminiscing about certain things, and so on.) It's very hard to explain what I mean. But the line is drawn in so many places when it comes to issues like this, and that's what makes it so confusing at times. Age, to me, is just a number anyway, so my philosophy is to enjoy whatever age you are, keeping as active as possible and realizing that it's never to late too follow our dreams.