Societies, Bayard suggests, maintain what he calls a "collective library," meaning a virtual collection of books that the culture is familiar with. (You know, the way most Americans are familiar with characters from The Wizard of Oz without having read the book or maybe even having seen the movie.) One of his points is that sometimes a book is more significant for its relationship to other books in the collective library than it is for its own content. If individuals understand or at least know about the book's significance, they can talk about the book in that way. And it would be very legitimate for them to do so.
Imagine, if you will, that it is the year 2057. A turn-of-the-century series of children's books about a kid named Harry Potter is oh, so yesterday. No one reads them, but everyone knows about Harry because of all kinds of literary references, movie references, maybe some song references. Harry Potter, though not read, is part of the collective library.
Say you are a graduate student in the year 2057, and you wouldn't read any of the Harry Potter books on a bet. But you are aware that the appearance of Harry Potter in the 1990s brought masses of adult readers to children's literature, encouraged serial novels, knocked problem novels off their pedestal in kidlit, and popularized fantasy. Knowing the significance of the series in relation to other books is arguably as important as anything in the books and certainly gives you something to discuss if Harry P. comes up when you're trapped in your advisor's office. In fact, there are probably many books we should know something about even if we haven't read them.