Gardenias are mostly tropical shrubs. The one you tend to get from the garden centre is G. jasminoides. They're not the easiest things to grow....they need lots of bright light and high humidity, they must never be dried out...that could be another reason for bud drop and should be watered with soft water (rain water). They dislike draughts and central heating and as mentioned, hate being moved around.
They're also very prone to red spider mite and scale insects.
I managed to keep one for 3 years indoors once....don't ask me how !!!!!!