Were any LLMs trained on Google books?

Question

An important limiting factor on the performance of large language models, is the amount of training text available. Of course, using e.g. the Gutenberg archive of public domain books is an obvious thing to do. But the majority of extant books are in copyright.

Over the last couple of decades, Google has scanned something like forty million books. Most of these are in copyright, so there are legal obstacles to making them available for download. As far as I know, this is the largest collection of high-quality text in existence. They provide online access to it on a very limited basis. I haven't heard of them ever giving or selling a complete copy to anyone; not sure they would have legal permission to do so.

Has any LLM been trained, either by Google or anyone else, on this corpus?

Answer 1

Were any LLMs trained on Google books?

We don't know. Not all LLM creators disclose their training set, but AFAIK only Google has access to all Google books, and Google didn't disclose if they trained their LLMs (LaMDA, PaLM, PaLM 2, etc.) on Google books. E.g., the PaLM 2 paper just states:

The PaLM 2 pre-training corpus is composed of a diverse set of sources: web documents, books, code, mathematics, and conversational data.