I would say not (without knowing the full facts its hard), unless you have signed a renewed contract agreeing to the terms, are you a member of a union? if not, it might be worthwhile contacting one to join.
Depending on the work you do it should be stated in your current contract that you get bank holidays off, and to change that he would need to change your contract. i would say agree to nothing and sign nothing until you have taken legal advice most places can change your contract but its normally done through mutual consent, if its a large oraganisation that recognises unions then they would have to consult the union before any changes were made to your contract.
does this affect just you or is it going to affect a few people? if your not in a union then maybe a trip to the local C.A.B would be worthwhile.