He only has parental rights, which include the right to intervene here, if 1) the baby's birth was jointly registered by you and him [ this applies after Oct 2003] or 2) you and he had an agreement whereby he had parental responsibility 'a parental responsibility agreement' or 3) a court has ordered that he have parental responsibility.
Otherwise has no standing in this or any other matter relating to the child because you have sole parental responsibility, as the mother.
If he has parental rights he has to be consulted.If he refuses to agree then he has a right to be heard. It is extremely unlikely that the court will say that a Canadian mother of a baby is to be prevented from taking the child to Canada .The child has to be with its mother, so where she goes the child goes. If she were manifestly unfit to be acting as mother ( e.g. practising drug addict always out of her mind) then obviously the case would be different.