It had nothing to do with external.
It had to do with internal... mating....
PHENOTYPIC AND GENETIC EFFECTS OF HYBRIDIZATION IN DARWIN'S FINCHES. - PubMed - NCBI
"Morphological consequences of hybridization were studied in a group of three interbreeding species of Darwin's finches....In agreement with expectations from a model of polygenic inheritance, hybrid and backcross classes were generally phenotypically intermediate between the breeding groups that had produced them. Hybridization increased additive genetic and environmental variances, increased heritabilities to a moderate extent, and generally strengthened phenotypic and genetic correlations. New additive genetic variance introduced by hybridization is estimated to be two to three orders of magnitude greater than that introduced by mutation. Enhanced variation facilitates directional evolutionary change, subject to constraints arising from genetic correlations between characters. The Darwin's finch data suggest that these constraints become stronger when species with similar proportions hybridize, but some become weaker when the interbreeding species have different allometries. This latter effect of hybridization, together with an enhancement of genetic variation, facilitates evolutionary change in a new direction."
Which is as you noted, it's greatest weakness, because it does focus on the external, and the pitiful weak and harmful effects of mutations....