I think Andy that when someone says to me 'it is a driver's car' - what they mean is that the driver can exercise his skill as a driver, push the car to it's limit on bends but importantly know the feeling of that limit. He / she can use the acceleration to good effect and enjoy some serious, and still legal, progress on the open road.
The opposite description would be a car that you just drive along in without drama, much performance, much good handling charecteristics but still safe and legal.
A drivers car in my view is something like a Suburu Imprezza and the opposite would be a Morris Minor 1000.
The problems arise when a non-driver (as discussed) tries to drive a drivers car and ends up out of their depth, and often in a ditch.