It's just one of those words we use to fill in space while we formulate the next sentence we're going to say. The alternative is to leave big silent gaps in our conversation, except for those few who are fortunate enough to be able to rattle on without repetition, hesitation or deviation.
Yes it's irritating, but less irritating than er, er, er. Unfortunately I also find it less irritating than the radio programme to which jno refers. The habit doesn't make Frank Bruno any less interesting to listen to; know what I mean?