I would think that the recoil is actually backwards (action and reaction), but the force acts on your elbow, causing it to bend and sending your hand/the gun upward. If you hold it sideways it will again act on the elbow joint, appearing to go sideways. I could be wrong.............
The centre of gravity (the greater share of the mass) is below the barrel and recoil forces the barrel around the centre of gravity, the resistance to backward motion provided by the hand holding the gun adds to the upward or sideways turning motion. In some guns part of the recoil is absorbed in a line behind the barrel isolating this force somewhat above the centre of gravity reducing the tendancy to rotate.
The recoil is in the opposite direction to the bullet and along the same line as the barrel. Because you hold a handgun below the barrel the force of the recoil is along your arm but perhaps an inch or two above it. This causes the recoil to move the gun up. If you hold the gun sideways, it will recoil out to the side. Example: Do a 'thumbs up' with your right hand. Use your left hand to push the fingers below the right thumb and your arm will move backwards = recoil in line with your arm. Now stop pushing the fingers and push the thumb and your hand will pivot up = recoil slightly above the arm.