It is not just something that you and the kids' father need to sort out - it's for them to be involved too, especially at the age of eleven. If your ex does apply for parental responsibility, then the court will take the children's wishes into account when making its decision.
As for hearing it every year, perhaps if you did sit down and talk about with them - even if that's only the eldest, then at least they would know what's going on and, seeing you a little more chilled about things, may just decide for themselves that it's really not worth getting into a stew about themselves. You'd be amazed how sensible and grown up an eleven year old can be about these things. Better that than the kids turn into bitter adults remembering that their parent(s) never told them anything - believe me, that can take a lifetime to shake off.