It actually depends on the variety of magnolia you are growing. Here, in the U.S., magnolias can come in about 60 different varietals.
Since most magnolias here are pollinated by beetles, rather than flying insects, like bees, it's possible that the appropriate beetle species hasn't been present. If the flowers aren't pollinated, the fruit won't form up.
Secondly, while many magnolias are capable of pollinating themselves, some require another tree in the vicinity for the pollinization process. Also, keep in mind that most varieties take nearly 20 years to flower. Perhaps your's is such a tree.
You're probably aware that the magnolia is one of the most ancient of trees in existence. Probably originated some 60 millions of years ago... (It's also the State flower of Mississippi, here in the U.S.)