2sp - just re-read your post, my apologies for 'skimming' it - of course that this the point we are both making.
I can't remember this chap's name, but when ever i see him on TV - he is on Have I Got News For You - he is always the one whose remark kills the humourous exchange because it is too left-field to be actually funny.
And that is what I feel he is doing here - hey, the notion of 'fighting' a baby - that's funny, right?
Er, no, not really, it's not.
'Playing' with your baby, which is what he is doing, that's entirely different, so why not just call it that, and avoid looking for humour that isn't there.