Say all four got through and they all shouted out answers at the same time, it would be bedlam. Even with two people at the end, what would happen if they shouted out different answers, one right, one wrong. It makes complete sense that they have to buzz. There is not the same need for the Chaser to buzz as he is on his own.