If the collection agency doesn't accept your payment plan, all they can do is to apply to a court for a CCJ to be issued against you.
If you didn't respond to the court action the CCJ could insist that you repaid the full amount immediately. However, as long as you provided the court with details of your means, the CCJ wouldn't require you to pay at a rate greater than you could afford. (i.e. as long as you cooperated with the court, all that the collection agency could get from going to court would be a court-backed repayment plan which you could afford. For that reason they won't rush to take court action against you; it's far simpler for them to just agree a repayment plan with you in the first place).
If the collection agency did decide to take you to court, and secured a CCJ against you, then you'd still have nothing to worry about as long as you kept up the repayments ordered by the court. (If your income fell even further you'd be entitled to ask the court to reduce your payments).
It's only IF a CCJ was issued against you AND you failed to comply with it that you'd need to really worry. It's at that stage (and NOT before) that the collection agency could apply to the court for an enforcement order (allowing them to send in the bailiff's or to take the money directly from your pay).
In summary then:
(i) it's probably unlikely that the collection agency will take you to court (as they'd only end up with a repayment plan similar to what could be voluntarily agreed to anyway) ; and
(ii) even if they did, you'd still not have to worry about bailiffs (or other enforcement action) unless you failed to comply with the terms of the CCJ (which, as I've said, would enable you to pay at an affordable rate).