If he has a right of access then you should not obstruct it (probably hence why you can't fence it off) even if it's on your legally owned piece of land. Have you checked if a right exists?
Check your title document (should be with the solicitor you used to buy the property or your mortgage lender - the latter may charge for a copy though).
The easiest way, provided your property is registered at the Land Registry, is to download a copy from the Land Registry for �3.
http://www.landregisteronline.gov.uk/
It should tell you if part of your land is subject to a right of way for someone else and may give an indication as to whom.
More than likely , considering the land, you each own the bit behind your house and have a right of way (which may be restricted, eg on foot, so check the deeds in case he is not allowed to take vehicles up and down) over the bits forming the lane in order to reach the nearest public highway.
It may be that there are provisions for maintainance of the accessway and provisions for contribution (very common in old deeds when a lot of accessways were not adoped by the council who maintained them).
If other vehicles use it could all users agree between you to share the cost of paving it or putting down a more suitable for vehicular access it vehicular access is in fact allowed (check the deeds as above).
If the right of way is by foot only (make sure this is absolutely correct though) you could maybe put something up which does not obstruct pedestrian but does vehicular access.
Please think practically though, don't go overboard and start a war, try to resolve any issue without getting into a dispute as you're all neighbours afterall.
Could you have a general chat with him or call a general meeting with the owners an