He applies to the Land Registry to remove the trecord of your having title. Assuming that the Land Registry doesn't think the entry made by mistake then, in the absence of agreement, the case goes before a tribunal of the Land Registry. I'm involved in one personally at the moment, where someone is claiming that the Land Registry should show them as owner by virtue of their adverse possession of over twenty years. The land is shown as mine on the Land Registry record of title.The procedure is such that it's possible for the matter to be decided on written submissions from both sides. I don't see that the procedure and practice would be different in a case where you are recorded as having title by virtue of adverse possession.
I've a feeling that claims for adverse possession are recorded by the Land Registry without strict legal proof of it. In that event it should be fairly easy for the original owner to recover the land, because it's quite difficult to establish adverse possession sufficient to give title.