No, there is no rule!!!
Your neighbour is selfish, and needs to get a life.
Please continue to feed the birds.
I have a similar neighbour who considers my washing line to be a nuisance, and tells me often.
He's one of these "la de dah's" who considers washing lines to be obstructive, and reckons it spoils his view.
I'm sorry but you get these selfish gits wherever you go, and THEY are the trouble, not you.
Unless your feeders are attracting large gulls, crows, or similar, you are well within your right to feed the birds, and your neighbour should like it, or lump it!!!