It's not trying to keep it in circular orbit. It's just the path taken in curved space.
Imagine the ball going around on the rubber sheet with the a dip in it. It doesn't necessarily hold a circular path, it builds up speed approaching the dip, shoots past, then gradually slows down as it goes up the potential the other side, and then turns around and approaches again. Eliptical orbit. Of which a circle is just a single special case.