I think if your new employer raises it -
you tell them what you feared would happen, happened.
This is not uncommon - my brother was invited to an interview and told the only reason why we are interviewing you is to tell you not to use one of your referees, [ who had agreed to support him in his application and then written a truly crap reference ]
When I was on an interview panel, I ripped open candidate X's reference and the writer said I dont know why I am writing this: he has accepted a job elsewhere. [He hadnt]
and for a third job - I commented to the panel as an interviewer, I spoke to Y the referee and he was less enthusiastic than I thought he should be. Has he written a cross-eyed reference sort of like: if you get candidate A to work for you, you will indeed be fortunate ? HR said yes, and I said we dont have to look at references do we ? [we didnt]
so good luck