of course he should be allowed to take her away! are you seriously suggesting he will never have her overnight, or want to see her away from your house? he has as much right as you do to have the child live with him, and could fight you for that if he was inclined to do so.
Is he on the birth certificate?
And why should he go to court for access? why not make it easier on your daughter and just allow him to see her without having to go through the courts.
As for living in a rough area - did it bother you when you were sleeping with him? If he's moved there since, well he is still the same person no matter where he lives, surely you cant be shallow enough to think that just because he lives somewhere you consider rough, he would let her come to harm?
The long and short of it is you chose him to be the dad, and no matter how unpalatable it seems you have to put up with being tied to him forever, including where he choses to live