Coming into your garden and barging into your house uninvited is NOT on. Your neighbours maybe able to lower the fence - or even put a gate in, but surely you could put a padlock or bolt on it from your side? The best thing would be to approach your neighbours in an orderly way, and say that whilst allowing the children to play together, which might be a "nice idea", you also want your privacy, and don't want chilren having access whenever they feel like it. Sorry, but you have to stand firm.