If a balloon ascends high enough, the atmosphere outside thins to the point where the mass of the air the balloon displaces no longer outweighs the mass of the ballon and its helium and it can therefore no longer float.
johnah is the most correct. In fact helium atoms are small enough to pass through many glasses as well as rubber this is why good helium ballons (for air ships for example) are made with a foil lining to reduce this diffusion. The capacity of helium to pass though glass causes problems for work at very high vacuums where a window is required. special glass is needed to stop helium diffusing though the glass and into the vacuum so ruining the experiment.