With the vast number of wars, conquests, skirmishes, and the occasional flat out slaughter that punctuates the Dark Ages, people who are different/foreign are viewed with suspicion. But the English have a special place in their heart for a few peoples in particular:
The Saracens
The Crusades are all about reclaiming the Holy Land from the Saracens, who today are known as Muslims. The history is long and bloody, and people likely know someone who was killed by a Saracen.
The Scottish
Durham's angry northern neighbors often bring bloodshed in their wake.
The French
Oddly, as often as English Kings are related to French Kings, they keep going to war with each other. The French were considered untrustworthy cowards at best by the common folk.