I agree with ludwig's last post. Fear is often born of lack of understanding - the "if it's different, it must be wrong" frame of mind.
I would have held that view myself at one time until I started exploring comparative religions and cultural viewpoints as a personal interest, and gained a better understanding myself, and started to move in a culturally diverse society. I hope these days I am not prejudiced on the basis of lack of understanding.
Having said that, the fact that the bombers etc who are high profile are (mostly) from an extreme branch of Islam does the rest of the Muslim world no favours at all, and leads to all Muslims being tarred with the same brush.
I fear that the Baroness is right in what she says, but that still doesn't mean the prejudice which may have become socially acceptable is any more correct than it ever was.