This is a game of knocking on doors and running away. Wikipedia says it is because the front doors of council houses were stained a ginger colour, but this seems unlikely.
Whatever the origin, I have a theory that children who play this game, grow up to be home-delivery couriers.
The phrase 'knock down' meant, as long ago as the mid 18th century, to knock at someone's door - innocently! - in order to get him to come downstairs and speak to you. Clearly, children then just took that as the basis of their naughty game's name.
Maybe - if he then became a regular 'tormentee' - the very first person the trick was ever played on had ginger hair and hence, "Let's go and knock down Ginger again!"