When the sun is lower in the sky (as it is at the poles) a given amount of energy from the sun has a larger area to cover. That's the principle reason why the temperatures at the poles are lower than at the equator. Distance from the sun has nothing to do with it. For the same reason the energy provided by the sun early or late in the day covers a larger area than at noon so the sun seems to "lose its power" as the sun sets.
As for distance from the Sun - The poles on average are less than 3000 miles further from the Sun than other illuminated parts of the Earth . . . rather insignificant compared to the 3 million miles difference in distance the Earth experiences annually.