I hate Microsoft Word, precisely because it's always doing things like that!
While I'm sure that there must be a 'proper' solution, my experience of such matters is that it's usually simpler to start again. i.e. create a new blank 24 page document (with page numbering) and then copy and paste the text from the original document.
That (extremely basic) method might take a few minutes of your time (especially if you've also got headers and footers to copy across) but it will probably still be a lot quicker than hours of mucking about seeking a 'proper' solution.
Chris