Yes, it's all about conservation of energy. If the universe is a closed system then it always contained, and always will contain, the same amount of energy. This energy can exist in various forms but only matter can have a temperature - the spaces between stars, in so far as they are truly "empty space", are neither hot nor cold. In terms of the question, the universe, in so far as it can be said to have a temperature at all, is as warm as it is, period. By definition, that "temperature" includes the radiated energy from stars.
Entropy is the tendency of the energy content of a system to "level out", so that eventually the "temperature" of the universe will be the same everywhere within it. This means that areas that are now colder than the average will become warmer, but the "temperature" of the universe as a whole will be just the same as it has always been.