Pink has a good point of course. That's why some of these situations are so difficult. Of course something needs to be done fairly quickly if you believe that you have good reason to worry, but I've also spoken to youngsters who were looking forlorn because their parents'd left them at nursery all day.
I totally agree that the child must come first. If you suspect that neglect is really happening, then all the above suggestions might be worth a try. All I'm saying is, that if the situation WASN'T to be quite as you're seeing it, then removing the child from his mum could cause more damage than a little, however good your intent. It'd also more than likely damage relations between your son and the child's mother, making access more difficult or unpleasant. However, you know more about this situation than we do, and, again as pink's just said, sometimes people leave things too late. Perhaps a Child Welfare Officer might be given the facts as you know them to be, and then hope that the professionals can sort something out.