If it were me - I would ignore the teacher's comments but for my own peace of mind I would make a doctor's appointment and his view, ask to speak to a paediatrician, if you wish.
I would do this, without my son, being present, just to get the doctor's view. You could also speak to the Head Teacher, voicing your concerns and how upset you are about these comments.
The teacher has not right to INSIST you see any specialist - it is down to you.
I actually had an Art Teacher who wrote a very nasty letter about my son to me because he was not working well at Art and would not get his GCSE in Art. At the time he was taking 8 GCSE in English, Maths, etc etc, He was in all the top sets, went on to College, Uni and is now an accountant - so much for the Art !! and the nasty Art Teacher.
Hope this helps. Above all - try and keep your feelings, and hurtful comments from teachers etc away from your son.
Maybe teacher has a chip on his/her shoulder.