I think the obvious thing to point out here is - IT'S A CARTOON! You must suspend your belief in reality for 20 minutes.
Is there any logic as to how Stewie can shrink himself and enter Peter's testicles in a space ship, or Peter can recover form a stroke by entering a shop etc? It is just taken that Brian is a member of the family - with Stewie, all the family can hear is baby babble, apart from Chris - I think he is the one that is able to mostly understand Stewie.