Motoring law doesn't actually refer to many specific situations. For instance, there is no specific law which states that drivers can't drive on the right-hand-side of the road. Neither is there a law which specifically states that drivers can't tail-gate another driver on a motorway, while doing a crossword. In such situations the police, the Crown Prosecution Service and the courts have to decide which of the various generic pieces of legislation apply. (e.g. they would have to decide whether the driver was simply driving 'without due care and attention' or whether the more serious charge of 'dangerous driving' should apply).
The Highway Code isn't law. It's not directly an offence to transgress the rules which appear in it, but a court could take breaches of the Code into account when seeking to determine whether an offence had been committed.
In terms of criminal law, you did nothing wrong. (Jay-walking isn't an offence in this country except where special rules apply, such as on motorways). There's prima facie evidence, from your account, that the motorist was gulty of at least 'driving without due care and attention', if not 'dangerous driving'.
If the driver had collided with you and your children, it's likely that a civil court would rule that well over 90% of the liability of the accident lay with the driver but it's possible that a small amount of liability might be attributed to you because of your failure to take sufficient care when crossing.
Having got all of that out of the way, I'll return to your actual question:
Because the law makes very few references to specific situations, and because the Highway Code isn't law, there's nothing in the statute books which gives either of you the right of way. However criminal law obliges the car driver to take due care and attention, while civil law expects both of you to exercise a 'duty of care' both in respect of your own safety and