I thought she said it to all of the guests, meaning that they shouldn't worry about leaving in the correct order according to rank and status etc. but just all leave as quickly as possible. I could be wrong though! x
Clem is right - Lady Macbeth was addressing all of the guests. JimJools may have been confused by the fact that the immediate answer to her request was spoken by Lennox - a nobleman, not a servant - but her object was to get everyone out of the house before Macbeth said something that would prove that he was a murderer.