"I'm really getting the feeling that they're a bunch of bullies. Seems ridiculous all the fuss you have to go through just because you don't want to watch TV."
Precisely. They assume every non-licence holder is an evader, and will send you regular, ever more threatening letters designed to intimidate you into buying a licence.
Their 'inspectors' are merely salesmen employed by Capita plc, and they're on a sliding scale commission - max of £20 for a direct debit sign-up.
Despite the quasi-legalese in the letters, they have only the same powers as a double glazing salesman. If you get a visit, don't confirm your name, and ask them to leave forthwith.