You could well be right, but I don't see what you can do about it, except to lodge a grievance perhaps on health and safety grounds.
If all your colleagues are of the same mind, next time you are short staffed due to the granting of unpaid leave, you could all do the same - go home, taking unpaid leave. The boss says he can't refuse, after all.
Or be unable to finish the day's work, needing overtime, if your work is of that nature.
However, your boss may well be of the opinion that a member of staff who has been refused unpaid leave may simply go sick, or be non-productive in the workplace that day.