It depends on the context. I would have thought that the answer is usually yes, since it's extremely patronising. It's usually followed by a statement which the person speaking thinks is obvious, and which only a "dumb guy" would disagree with. It also includes the inference that you must be a dumb guy, since he wouldn't be saying what he's saying unless you disagreed with him...