Whilst ebooks do seem to average around the £4 mark, many are cheaper and lots of classics are free because they're out of copyright.
How many times, though, do we buy books, either new or secondhand, and never read them. And how often do we buy or borrow a book, read it and then love it so much that it hurts to part with it? I can answer 'quite a lot' to both of these questions and I suspect I'm not the only one. On an ereader I can keep them for as long as I want without taking up any extra space. I think what the manufacturers intend to show by this selling point is that the capacity is pretty much enough for anyone no matter how often or seldom they read.
Yes, Amazon (and presumably the providers of Sony format books too) can withdraw their titles at any time, because what you buy is not the actual content, but the licence to it, for which the author receives a small royalty. However, I doubt they would or could do so without very good reason.