Wracking brains - of which there are not many - but I don't really understand the question exactly. I would have thought a novel is literature, do you mean like a classic?
Literature may just mean reading-matter in general, but - in the more restricted sense to which you refer - it implies writing with a claim to special greatness because of its beauty, depth, emotional effect, grandness of theme, characterisation, symbolism and so on.
Thus, James Joyce's novel, Ulysses, is literature whilst a Reacher novel by Lee Child is no more than a darn good read!
I remember at university we spent a lot of time discussing this question - and I'm still none the wiser. I seem to recall the idea of 'literature' came about during the early nineteenth century as English literature began to be taken up as serious academic study. Along with that came the idea of a literary canon, and one of the chief voices on the matter was one F R Leavis, whose ideas may just inform you (or not) on this.
Oh LeMarchand - I just so disagree. Sure, there are a few works of so-called literary merit that are difficutl to read, but by far the great majority of great works of literature are accessible for the very reason that they speak to everyone.
If by "great works" you mean those that have stood the test of time and gone on to become "classics" (eg Dickens, Austen etc etc), I would totally agree.
My beef is with the so-called modern "literature" that wins awards, most of which is pretty unreadable. I spent a long time in the book trade and (for instance) the Booker shortlist always seemed to consist of a bunch of books that would only sell after the list was announced and, even then, only the eventual winner would ever sell in significant quantities. I always wondered how many of those purchases translated to the books being read, too.
For example: it's not a Booker winner, but how many people actually read "The Satanic Verses"?
A good example of the lack of worth of so-called "literary merit" is the Doris Lessing/Jane Somers affair, where established author and critical success Lessing found it very hard to shift a book written under a pseudonym.
I would agree with LeMarchand - John Banville's The Sea was quite probably the most boring book I have ever read. Modern books defined as literature often do not either entertain (not necessarily a pre-requisite but helpful) or even have you caring what happens to the characters.