If it's a shared garden or space, and if he hasn't been staring through your windows, then no, I don't think he is invading your privacy. I understand it can feel like that though. As long as he's not there all day every day you might just have to cope, or draw your curtains while staring at him.
If he isnt there all the time and is just planting some flowers, he probably wants the garden to look nice. If he is looking in your windows that is a different matter.