The 'right' to divorce requires only the marriage has irretrievably broken down - an unfaithful spouse is valid grounds only if the 'wronged' party finds it impossible to forgive and continue with the marriage.
Your husband cannot be punished in divorce by making the terms more favourable for you. You will have the same rights as any other divorcing wife, regardless of who is 'in the wrong'.