Yes, literally, it's for wearing when combing your hair (or having it combed), but translates now as 'dressing gown', which, after all, is what a woman would wear when doing her hair in the morning.
French is good for literal descriptions like this. A boudoir is 'a room for sulking in ' (from bouder 'to sulk'). Ah, those French women !