I've been thinking about this question very hard for about ten minutes and, obviously, not got anywhere.
I think that you have to invent the ground rules, the axioms, of mathematics, certainly (although even this might be up for debate). Beyond that I think it's a journey of discovery. You follow the axioms where they lead you. Or, in the case of most of theoretical physics, you make it up as you go along and hope that someone else clears up the mess a few years later. But even then, the making-up process works, and leads to useful results.
So it's probably a mixture of both.