No I dont think it would be giving in to withdraw.
Pointless gestures like knee taking, that are affiliated to the MArxist group BLM anyway, will get us nowhere. Kick it Out was a good campaign and we need more like that, totally devoid of any political connection, and something every right thinking person can get behind.
Just think, if sufficient countries pulled out then the World Cup that year would be pointless and indeed worthless to any 'winner'. Not only that, from those left in we could start to understand how certain countries believe.