Bumble bees make their nests in holes in the ground- often disused mouse nests. So if there is a hole there, tht could be it..- the hole may not lead directly in- they often like a tunnel down into the main nest.
They aren't agressive, and will do a great job of pollnating our plants too.. Lots of people would LOVE to have a bumble nest as they are getting quite rare.
However if it is a flying ant nest (black bugs biger than normal ants and with wings), then feel free to boil a kettle and pour it over the nest. Horrible things that will plague you!