Was the driver reversing out of his drive or driving out frontways? Not sure if it makes a difference, but for me, if the car wanting to return up the road was reversing along the road, he was travelling in the opposite direction from which the driver coming out the driveway would expect to see a car, so whilst they should both be careful, if you are effectively travelling the wrong way up the road, i think that the onus is on you to make doubly sure that it is safe. However, if the car coming out of the drive was driving out, he should have been more aware of what was happening. If he was also reversing then they should both have been taking extra care.