St Edmund was indeed 'buried' there, but that is not where the 'bury' part of the name came from. There are, in fact, quite a few places called 'Bury' around England - including the one called just that in Lancashire. All of these took their names from the Scandinavian word 'byr', meaning a dwelling or settlement. So St Edmundsbury - now Bury St Edmunds - is just 'St Edmund's Place', really.
Interestingly enough, in England the suffix "-bury", as in Canterbury means "town of". In Thailand, the suffix "-buri" as in Kanchanaburi, means the same thing. Apparently they both come from Sanscrit.