We've been swarmed twice in recent weeks, fortunately we are in a position to relocate them into hives.
They may relocate as a bird box is no use to them, you could find a local beekeeper on this site
https://www.bbka.org.uk/about/local_associations/find_an_association.php who will collect the swarm, take them away and house them.
If it's only dozens it could be the recce party, a full on swarm can be as many as 10,000. If the bird box is high up, which presumably it is they shouldn't give you any bother. Most swarms are friendly but you can get the occasional bad tempered colony.