The issue with unglazed tiles is that the mortar sticks to it and indeed sinks into the surface that touches the tile.
The way I would do it is with wet mortar, strong (say 1:4), making the whole surface wet first (to thoroughly wet the substrate under where the grout lines will be). Yes, I know the tiles will also get wet.
Then with wet mortar and a pointing towel (sharp point, about 4 inches long diamond-shaped blade) I'd run the mortar into the gaps. Then leave it to go off a bit - probably 30 minutes, before brushing any that has got onto the tile edges off.
You are perhaps spending too much time trying to 'wipe' the excess off at the time of doing whilst it is still wet. That makes things worse perhaps you wipe wet mortar onto the tiles where it will mark. Wait until the excess has dried but not gone hard - then it brushes off.