Depends on your interests, but I'd suggest spending as much time in Durham as you can.
Stuff to do in Durham:
* wander round the cathedral
* walk along the cobbled streets toward the river
* wander around the river banks, down below the cathedral
* Admire the magnificent views of the cathedral from various vantage points around the city
(I'm not selling this well, but it's worth spending some time ralaxing in Durham.)
Manchester is a big city, much like any other, but Durham has a bit of an edge, if you like views, out-doors-y stuff and enjoy trekking up and down hills and quaint olde-worldy cobbled streets