Looking at the passage in context it is speaking of the men that Jesus actually spoke with and not people of all times. This is seen in the passage, but also verified by Jesus word later on.
11 And now I am no more in the world, but these are in the world, and I come to thee. Holy Father, keep through thine own name those whom thou hast given me, that they may be one, as we are.
12 While I was with them in the world, I kept them in thy name: those that thou gavest me I have kept, and none of them is lost, but the son of perdition; that the scripture might be fulfilled.
(Jn. 17:11-12 KJV)
In the passage Jesus said that He should lose none. The word "should" indicates intent not conclusion. We see from Jesus' words that one of them, the men the Father had given Him, was lost. The word "given" is a past tense word, thus indicating a completed action. When Jesus gave His prayer, in John 17, the men had already been given to Him. That this is a past event is further illustrated by Jesus word later on in the same pray.
20 Neither pray I for these alone, but for them also which shall believe on me through their word; (Jn. 17:20 KJV)
Here Jesus makes a distinction between those the Father had given Him and those who would believe through the preaching of the apostles. From this we see that those who would believe are not those who were given, thus they are two different groups.
Yes, Jesus mention two groups, but acknowledges both groups as having been given to Him by the Father.
v24 includes those who will believe on the testimony of the apostles as being given to Him by God.
"as Jesus says "that they also which you gave ME may be with Me."
The 'they also' is the all believers group.
This is distinct from where Jesus mentions the apostles in John 17:6
I have manifested Your name to the men whom You have given Me out of the world. They were Yours, You gave them to Me, and they have kept Your word.
Jesus Prays for All Believers
20 “I do not pray for these alone, but also for those who will believe in Me through their word;
21 that they all may be one, as You, Father,
are in Me, and I in You; that they also may be one in Us, that the world may believe that You sent Me.
22 And the glory which You gave Me I have given them, that they may be one just as We are one:
23 I in them, and You in Me; that they may be made perfect in one, and that the world may know that You have sent Me, and have loved them as You have loved Me.
24
“Father, I desire that they also whom You gave Me may be with Me where I am, that they may behold My glory which You have given Me; for You loved Me before the foundation of the world.
25 O righteous Father! The world has not known You, but I have known You;
Besides that, to be consistent, stick with what Christ says in other places when referring to any persons who come to Him as being given to Him by the Father, not singling out just the apostles. Christ eventually says if He be lifted up, I will draw all men unto me, so then of course being glorified, Christ is of course also the Father and also does the same things the Father does. So then He is the one doing the drawing of men.
And of course all men will be drawn irresistibly to God and Christ for judgement on the Last DAY as He is sitting on the throne of His glory.
Matthew 25:30-32New King James Version (NKJV)
The Son of Man Will Judge the Nations
31 “When the Son of Man comes in His glory, and all the holy angels with Him, then He will sit on the throne of His glory.
32 All the nations will be gathered before Him, and He will separate them one from another, as a shepherd divides
his sheep from the goats.