I think that makes a big difference - the sort of people who are going. Whether they're the sort that will deliberately lead astray, or just a bunch of good lads just wanting a laugh.
I think you would be either very naive or stupid to think they were going to Amsterdam to ride their bicycles or wander round the tulip museum - no matter how 'decent' they appear to be at home. I guess some people are better at turning a blind eye to these sort of things than others. I would consider it normal to feel worried about it.
It makes me laugh, the majority of British people that go into the coffee shops in Amsterdam think it's a good idea to go straight for the strongest strains of weed on sale and just end up greening themselves right out!