Only you can decide this. You know what the legal position is - you need to give 4 weeks notice. Will your employer sue you for breach of contract? - probably not. It isn't worth it because it is a lot of trouble when the employee has decided to go anyway.
Will they give you a bad reference? They cannot give you an untrue reference, but they could say something like 'we find X not entirely reliable, for example he left our employment giving only one week in notice when he knew this this would probably put us in a difficult position replacing him at short notice'.