I think it's generally accepted that it arises out of the 'us and them', in-group vs out-group, modal thinking of tribalism. It seems that, in general, members of groups that are distinguishably different from those we identify with, particularly in appearance, tend to be treated as 'other' and seen as a threat and/or inferior. It's particularly true of cultures where certain groups have traditionally been seen as or treated as inferior. When you grow up picking up subtle behaviours clues from the media and the people around you (i.e. cultural biases), it's hard to recognise them, let alone control their influence on your behaviour.
Studies have shown that even people who say they believe they are not in the least racist often show discriminatory behaviour in practice, and show subconscious threat responses in lab tests, that belie their beliefs.