The full expression is "for crying out loud". I have always understood it to be a way of saying "for Christ's sake" without actually saying it, in a world where such a phrase was unacceptable. In the same way some people say "Oh, shoot" instead of another word starting with "sh" and ending with "t".