Some bits get bigger- some get smaller-
Erosion takes soil, rock etc from exposed places- the wind may take some away but a lot is transported away by rivers and dumped elsewhere.
Also think about the number of leaves and other plant life that has rotted down over the years and built up on the ground surface. I laid down a path of square slabs down my garden some 20 years ago in the form of stepping stones. The surface of those slabs is in places an inch or more below the surface of the surrounding ground now.