It's your life, your choices. I think physics is great and it would be (one of the) top of my list for subjects to study, but if you are just tolerating it,drifting along, and not doing well then maybe a change could help you.
If you need biology then you are presumably looking at a science career of some sort. Additional understanding of other science fields can only help.
Was there a particular subject you'd prefer and think you could catch up on ? Or do you think it might be better to try to get more involved, accept it as not being ideal for a couple of years (life is rarely ideal) and see if you can get a good science foundation to build a career on ?