I think that you have misunderstood the meaning of 'fittest' in this context. In an evolutionary sense, 'fitness' means differential reproductive success. Nicholas Copernicus (1473-1543), Isaac Newton (1642-1727), Caroline Herschel (1750-1848) and Florence Nightingale (1820-1910) were 'unfit', for all their long lives and their great achievements because they did not leave descendants. Mary Boleyn (ca. 1500-43), Prince Albert of Saxe-Coburg (1819-61), John F. Kennedy (1917-63), Elvis Presley (1935-77) and Diana Spencer (1961-97) were fit in an evolutionary sense, in spite of their comparatively short lives, because they had descendants to continue their lineages.
If you read The Origin of Species, you will find that Darwin never mentions human evolution or human races anywhere in the book.
This is like the passage in Edmund Crispin's detective novel Swan Song (published in 1947), in which an undergraduate says that Wagner's music was bad because Wagner himself was sexually immoral and dishonest with money. Another undergraduate counters this by saying that by the same argument Villon (a thief) and Wordsworth (vain) were bad poets, Michelangelo (a homosexual) was no good as an artist, and Gluck (who died of drink) was also a bad composer. Whether Darwin was racist or not has no bearing on the truth of his theory. You might as well say that Hooke's law must be false because Robert Hooke slept with his niece.