People have a lot of misconceptions about acquiring rights to land, it's nowhere near as straightforward as many people think.
Maintain, and evidence where possible, that he is using the land with your permission (is there any kind of written agreement in place or even other terms of him using the land?).
You might find this quite recent case interesting -
http://www.propertyla...sessionpyegraham.html
I remember a lot of buzz coming up about adverse possession when it came out and it coincidentally concerns letting land be used for grazing. Obviously there is a lot more complex legal argument and individual facts than in this summary but you may find it interesting and if you google Pye v Graham you will find lots more information.