Apart from having your bandwidth stolen, and illegal downloads logged to your IP address? If you've shared any of your drives or folders - they could well be accessed from outside as well.
However, the point is that as the law currently stands, you are the owner of the router so are responsible for it and the content going through it. It would be the police at your door, not theirs.
Even if they have no proof it was you downloading that illegal stuff, and can never actually prosecute you in court, do you want the hassle of all this?