Around 1542, when the phrase first appeared, "to go to pot" was to be cut up like chunks of meat destined for the stew pot. Such a stew was usually the last stop for the remnants of a once substantial cut of meat or poultry, so "going to pot" made perfect sense as a metaphor for anything, from a national economy to a marriage, that had seen better days. Early uses of the metaphor were usually in the form "go to the pot"
If someone has gone to pot then they are thought to have deteriorated or declined from their previous status. The pot here is the melting pot into which valuable pieces of stolen silver and gold were remelted. They had gone to pot never to re-appear again.