If you take the house off the market you will need to get the planning permission to build on the land before you will be able to sell the garden. No-one will want a piece of land unless it can be built on.
That will cost a reasonable amount of money in fees as you will need to design something to go on the plot. I suggest you do that and if it works fence off the land, sell your house and have the little house you're looking for built on the land.