It doesn't actually go anywhere - and that is why lost data is able to be retrieved by techie experts and agencies.
However, by deleting data you are instructing the pc to ignore it in searches and allow it to be overwritten - so that tiny bit of the hard drive or removable disk is used again.
Eventually, it would become impossible to retrieve data that had been deleted long ago.
When new data is saved to the disk it is not saved in one nice piece - it is used to fill the gaps that have been deleted, so can be scattered in tiny segments.
This is why it is so important to defrag your hard drive regularly - it keeps things tidy and speeds your pc up.