If you were caught (which, as I've said, is extremely unlikely) you'd be refused entry and probably put on the first available flight back to the UK, at your expense. If the only seat available was in business class, at several thousand pounds, you'd have to pay it. While waiting for a suitable flight you'd be held in custody (i.e. locked up in a cell or in a room in the immigration suite). There's no obvious reason why anyone travelling with you would be affected by your problems but possession of a valid visa (or the correct completion of an ESTA application) is not a guarantee of entry into the USA, so it's still possible that those travelling with you could be refused entry.
Attempting to enter the USA without the correct documentation isn't, in itself, a criminal offence. (The thousands of Mexicans who attempt to get across the US border every day aren't breaking an American laws). However lying on an ESTA application is. (It constitutes fraud). So there's a theoretical possibility that you could be arrested, brought before a court and imprisoned.
I repeat that I think that it's very unlikely that you'd have any problems if you tried to enter the USA under the Visa Waiver Program, with an ESTA. It's even more unlikely that you'd end up in prison. But your question asked for the worst case scenario, so I've given it.
Chris