Double yellow kerb markings should be supplemented by a '"No Loading At Any Time" plate but the absence of such a plate (or even of the kerb markings) wouldn't necessarily invalidate the order prohibiting loading (and, by extension, parking by blue badge holders). The courts have ruled, for example, that parking where there would normally be double yellow lines (even though those lines have disappeared due to resurfacing of the road) can still lawfully lead to the imposition of a fine/penalty charge.
So I'd advise not attempting to use your blue badge there unless your local authority has chosen to vary the normal rules about "no loading" areas (as they're permitted to do) so that blue badge holders can use them.