Hey Sally-ann,
People have always said to me to leave it a couple of weeks.
When my cats have had kittens though, we have always handled them for one reason or other after a day or so, and it doesnt seem to of done any harm.
We always handled them very close to the mum cats who have kinda of given us a look of 'we trust you but dont push your luck!'. None of them ever got stroppy or nasty, and we never let any one other then me or my partner touch them.
But it depends on your cat I suppose, all my lot are very affectionate and loving, and I even had one of them cuddle up to me on the sofa and had her 'waters break' all over my arm! When we tried to move her off the sofa she kept trying to jump back on for a cuddle, so in the end we put towels on the sofa next to me and let her get on with it!