It seems to depend on the sort of place. I have never been asked for id using a credit card in restaurants, but try buying a shirt in a department store and they want to see your passport. It is a legal requirement to carry picture id at all times in the USA, so best to keep passport with you.