It is not only the fact that the prisons are full. (Actually they are not, but they are near to full capacity).
When judges or Magistrates undertake sentencing they must have no regard for the fact that prison space is limited. (Imagine the stink if it was reversed - “we’ve got a quite few spare places available in the prisons so let’s send a few miscreants to jail who otherwise might not go there”). What they must have regard for is sentencing policy and in particular sentencing guidelines issued by the Sentencing Council. This organisation is allegedly independent of the Government. However there is no doubt in my mind that they are heavily influenced by the government of the day.
During the last few years the sentencing guidelines for many offences have been pushed down. Offences that previously would have attracted an immediate custodial sentence now have a “Starting Point” of a Community order. Longer jail sentences have been reduced. This is the way the government overcomes its failure to provide sufficient prison places. As well as this various schemes are in place to ensure that those sentenced to custody do not serve anything like the length of sentence passed down. It was not that long ago that prisoners received up to a third off for “good behaviour”. This reduction could be withdrawn or altered at the behest of the prison governor but this was ruled unlawful. The strange thinking behind this decision was that prison governors are not a “properly convened tribunal” and are not in a position to impose extra custody. (The reason I say the thinking is strange is because they were not imposing extra custody but simply reducing a privilege. But no matter). Now, however, all prisoners serving determinate sentences are released after serving, at most, half their sentence regardless of their behaviour whilst inside.
There is a lot more to the answer to your question. Not the least is the unusually high impact that some organisations who, for a large part, do not believe in prison as an effective disposal at all, have on government policy. (e.g. the Howard League for Penal Reform. Their motto is “Less Crime, Safer Communities, Fewer People in Prison). They do not, as you suggest, reflect majority public opinion but it is the public from whom the State has removed the option of dealing with those who commit crimes against them. In return the State is supposed to impose effect penalties on their behalf. But many people think that it simply does not.