You need to get your head around the difference between being a shareholder and being a director - they aren't the same thing.
Shareholders own the business and, because you each have one share, he owns 50% of the net shareholders funds.
However you are the only Director, and Directors run a business. So, subject to the Articles of Association of the company, you may be able to operate by taking a (large enough) income as a director of the business such that you make a loss each year and gradually run down the net shareholders' assets (funds). The shareholders own the net assets and you are right that the business cannot declare a dividend without him being entitled to half of it. Of course paying yourself a larger salary instead of a dividend may not suit your own tax circumstances. That would be one way of extracting income from the business without it involving him. I hope you've tidied up the release of him as a director with Companies House.
You really need specialist help to find a way of releasing him as a shareholder. Without knowing the detail I can't help with that.