There's no standard answer to your question because the access rights will (assuming that they exist) be defined by a covenant within the deeds to your house.
However, it's common (with terraced housing) for residents of the central properties to be allowed to walk across the rear gardens of neighbouring properties to get to their back doors. This access right is usually (but not always) extended to non-residents visiting the central properties (e.g. milkmen, paper boys, postmen and guests of the property owners). The rights normally only give access to pedestrians; driving a vehicle or riding a horse across your land is probably not permitted by the covenant.
As I've indicated, that only provides guidelines as to the type of access rights which your neighbour might have. Without seeing the deeds to your house, it's impossible to give a precise answer.
Chris