I had an interesting experience like you many years ago.
I drove 200 miles from London to Cornwall, on the way the engine started loosing power, by the time I got there it would not go over 80mph, on the way back it would not go over 70mph.
The next day it would not go over 60mph, later in the day it stopped running.
Called out recovery, they checked everything, sparks petrol etc, they eventually found out the rear silencer box was completely blocked by the fibre packing inside it, took the bonnet stay of the car and used it to pull out all the fibre, it started immediately and afterwards seem to go even quicker and with no extra noise from the exhaust.
Did replace it though.
A lot of the rear boxes don't have baffles they have a perforated tube running straight through, around it is packed the sound deadening material, when the tube corrodes and collapses the sound deadening gets pushed out of the back and sometimes piles up and completely blocks the silencer.