Even if the father doesnt have parental responsibility he still has a legal right to apply to the Court for an interim Order to stop you leaving the Country with the child, as he is the child's natural father. The Court will then ask both parties to put their case across and decide what is in the best interest of the child. If you can prove that you can maintain a routine of contact with between the child and the father then the Court may decide its ok for you to go