Most of the light which enters and is reflected by a raindrop does not contribute to the rainbow. Raindrops below the primary rainbow reflect this �extra� sunlight at random angles which are not partitioned into colour bands but only brighten the sky. Light which strikes a raindrop at an angle to sharp to form the colours of the rainbow are mostly scattered outward away from the observer.
To see the angles of refection within a raindrop see
Light Sky Under Rainbow
To appreciate the degree of contrast see also
Alexander�s Dark Band