As both criminologists and psychologists would point out, 'external inhibitors' (such as the fear of punishment, whether that be a prison sentence or eternal damnation) often fail to prevent criminal (or anti-social) types of behaviour. 'Internal inhibitors' are far more important. For example, the reason that most men don't rape women has absolutely nothing to do with either their fear of prison or their fear of damnation. It's simply that their internal belief system tells them that such actions are unacceptable.
While religions might claim to provide an inflexible set of rules (such as the Ten Commandments) which everyone should live by, most people who live in a largely secular society (as here in the UK), and who may be atheists or agnostic, still seem to acquire the internal inhibitors necessary to function within that society.
Or to put it more simply:
No, society would not fall apart. Indeed, it would be far better off without the rules of religion which lead to prejudice, persecution or segregation (e.g. against women or homosexuals). Society would be far better if we banned the mention of religion to young people until they reached adulthood. Then (free of the brainwashing that many children currently receive) they'd all fall about laughing at the utter stupidity of it all.
Chris