Don't worry about the fact that you are EXDirectory,as chas says it is all done by computer (in the UK,where they ARE allowed still,or from abroad,which is more usual).
The computer dials out numbers at random,when it gets a ringing tone it connects it to an operator or (as in your case) to a recording.If the phone is not answered in a short while the computer cuts the call off and moves on to the next call.It does NOT know what number it is calling,or if it is EXDirectory.
You can stop these (number witheld calls),but I think you can only stop the UK ones;but check with TalkTalk first before paying for this service.
You could also think about having a "Caller Display" service(if available with Talk Talk) this would show a dispaly on a small screen of the number calling you,and if you see"Number Witheld" you can chose not to answer the call.