It was a risky situation but it turned out not to be an issue after all. I think you are simply nervous at what you imagine might have been.
Perhaps you can bring up the subject of health & safety, and being the only one there, with your boss when you talk. But I don't know the situation and more staff on may not be feasible. But maybe something can come out of the talk that can be done to make staff more at ease.
Otherwise it is a part of the job and you could always keep an eye out for alternative employment where you would feel safe.