Doesn't sound like something which should be done rashly or on your own initiative if possible, better to tread very carefully.
Who did you hear about the former manager from?
If this has been a previous problem then do management know about it? Have you spoken to your line manager about the situation? Maybe agree, with other senior management and HR as appropriate, the best course of action in line with company policies, employment law etc...
If nothing else it may well guage the kind of support you may get now and later.
I'd tread very carefully, especially if it is individuals/a team who may already have caused problems.
In the meantime, carry on doing you job how you are supposed to do it, be professional, consistent, fair etc... and give no cause for any warranted fault picking as much as possible (give them no legitimate reason to cause issue) and remain approachable as to any issues.
Try not to show paranoia or weakness (give them something to get a grip of) without beng overbearing or too defensive the other way and getting backs up.
Sounds much easier than it is though, granted.