My memory is a bit hazy, but as I recall, the song's writer was visitng a particularly poor area of the world, and saw a little pauper child carrying an even smaller child in a piggy-back. Since the older child had barely enough flesh on his frame to support his own weight, it seemed a miracle he could carry another child as well. When asked, the child explained with the touching and inspirational observation "He's not heavy mister, he's my brother." and so the lyric was born, and brotherly love was imortalised - such is the power of pop music.