Child benefit is awarded to the resident parent, that's like to be seen as you despite joint custody unless you tell them otherwise.
It's good that you listen to the kids and let them decide, but you would be better off with set days in place to give you all some stability, this way your ex may be able to arrange his work schedule whereby he doesn't work so late on the nights he has the kids. Assuming you don't live far from him, this should not interfere with the kids' social lives. If you are the one in receipt of the child benefit, he cannot take it off you - he would have to prove he's the resident parent, which (IMO) could only be verified by a court. The DWP will not just stop giving it to you then hand to him.