I'd prefer buying from a local greengrocer etc as you usually get better service but to be honest I usually use the supermarket. It's just so much more convenient to buy all your goods in one place.
To correct one of the previous answers, you can of course buy fruit and vegetables (also meat, fish etc) loose and not prepacked in plastic in the supermarkets.
I've bought at market stalls before but find you do have to be very careful. Several times now I've got home to find mouldy fruit that the stall holder has added from behind the stall.
Generally I'd say a supermarket can be the cheapest option but, as has been said, cheap is not always best (see my market stall comment as an example).