I have an 11 year old son and a 9 year old daughter and have been through similar. My son is mostly confident and in new situations makes friends very easily e.g. on holidays. However, he doesn't enjoy sports very much, and sings in a choir (his choice)so therefore regards himself as being "uncool",but I, like you have always encouraged him to be individual.
My daughter has always been very shy at school and never has a particular best friend. However, she is generally quite happy.
The school they attend operates a couple of very good systems at lunchtime. 1) They have a group of Year 6 children responsible for different games to play in the playground - they work on a rota system and involve as many children as possible. 2) Every day there are 2 older children assigned to "work" at the "friendship stop". This is an area that children at the school who haven't got anyone to play with can go to and the befrienders will play with them. These 2 schemes have been very successful and we very rarely have a child who feels isolated at play times. Perhaps you could suggest something like this be initiated at your childrens school (if they haven't got something already). You could voice your concerns to the teachers, so that your son and daughter could be kept a special eye on at lunchtimes, by the midday supervisors.
As for birthday parties etc., older kids don't seem to have as many parties as the younger ones, so I shouldn't worry. And parents that never invite your kids back to tea when you have had their kids time and again (and there do seem to be a few of these), just don't invite their kid any more.
I think, what action you decide upon must be based on how unhappy your children are. Always go with your gut feeling as their mother and don't be hesitant in speaking to the teachers/ headteacher - they are there to help.
Good luck and I hope everything works out for you and your children.