In general, anyone entering the USA under the Visa Waiver Program does not require 6 months validity on their passport but anyone who is required to obtain a visa (e.g. because they've been arrested or convicted of a criminal offence) must have 6 months validity on their passport before a visa will be issued.
However, there seems to be a specific problem which doesn't affect most travellers but will affect your mother: Entry under the Visa Waiver Program is only available to people who hold a
machine-readable passport. Such passports were introduced by the UK Passport Agency in 1998. Since your mother's passport was (presumably) issued in August 1997, it won't be machine readable, so she can't use the Visa Waiver Program with it. Neither can she obtain a visa because there's insufficient validity on her passport. So,
your mother must obtain a new passport.
References:
http://london.usembassy.gov/cons_new/visa/niv/ vwp.html
and
http://www.passport.gov.uk/news/news.asp?intEl ement=1009
Chris