The answer from sarah is, how do you Brits say? Spot on. I'd only add that the winds above ground level can be blowing at different directions at different velocities. It's not unusual for hot air balloonists to ascend and travel several miles in one direction, then climb or descend a few thousand feet and fly or drift, actually, in another direction. This change in wind directions create a shearing effect at the boundary of the two air masses which can cause turbulence as well.
On nice summer days, you may notice fluffly white clouds. These are known as fair weather cumulus, since they rarely produce any rainfall. As sarah notes, it will be bumpy below and at the same level of these, but will be smooth as silk above. They are caused by the uneven heating of the surface of the Earth producing columns of rising air...
Please don't worry about loss of contro/crashing. I've been a commercial airline pilot most of my adult life and flown most of the Earth's areas... rarely had a problem with weather/turbulence... though, there was that time 20 miles south of Mt. McKinley...