Worried mummy, if you want to make your partner legally her guardian then he can have parental responibility but I can believe the people telling you not to worry about him. He is her father. Not being on the birth certificate means nothing, he can prove this, and go to the registrar and have his name put on it and he will have as many rights as you. The more you exclude him, the more likely he is to take steps to get her as much as possible. Ask him about her schooling, tell her future schools to send a copy of her reports to him. Invite him to her birthdays.
I can manage these things with an ex who broke my arm, I think you need to grow up a little and think about how you are going to answer the questions your daughter asks about her daddy, if you succesfully bully him out of your lives. Will you lie? Then what happens if she finds out you lied? What happens if man of the moment leaves? Will you then change tact and start chasing the "old fogie" for maintenance.
Also if you have ideas of your partner adopting her, with her dad on the seen this will never happen. Oxford to Plymouth is a long way, twice a month is nothing to you and yet a lot for him to do. Sounds like he is trying to bond with his daughter. Imagine if the boot was on the other foot for just one minute.