It's because they hear the abbreviation 'should've' and hear "of" and not "'ve". They've probably never read or written it.
The same goes for brought and bought. I have a friend that says "I brought a chocolate bar", referring to the purchase and not the fact he took it with him.
Nothing annoys me more than 'H' pronounced 'haych' and not 'aych'. It's just incorrect. And so many people do it!
On a finer note, "James' car" implies there is more than one James. Its "James's", or even "Boss's", but too few people care about excercising good grammar.