Hmmm. That doesn't sound terribly satisfactory.
Do you know who does own the section of land over which you have access rights?
If it is not the council, have you inspected the land registry record of the land on which the council building is situated, to establish exactly what right of way they do have? The council land will be registered at the LR, the same as any other land - with the registered proprietor being Great Fopping District Council, or whatever.
Are we to assume this driveway is not tarmaced then?
I can't see how you can use this aspect in isolation to deny PP - but you could seek to have permission granted on condition that access is properly finished and adopted as part of the public highway (if that's what would be OK for you).
Is this garden centre something the council are doing themselves, or are they merely trying to sell and excess piece of land with OPP for retail development for a GC, to a commercial enterprise?