Researchers at China’s Tsinghua University an Zhipu AI have created a large language model (LLM) called LongWriter that is reportedly capable of producing text output of up to 10,000 words.

Current LLMs are not typically capable of producing very long answers the length of full books or manuscripts. Currently, the limit is roughly 2,000 words. The researchers suggest that this is likely because they are trained on short documents. As such, they team determined that if LLMs are trained using longer documents, they are subsequently able to produce longer documents.

Source: arXiv (2024). DOI: 10.48550/arxiv.2408.07055

To confirm this, the researchers first trained a 9 billion parameter LLM using a conventional dataset that included documents that were mostly under 2,000 words long. As such, when instructed to create text, the LLM couldn’t produce texts more than 2,000 words long.

The researchers then modified a traditional LLM via a pipeline dubbed AgentWrite to decompose training material into so-called subtasks as it was processed. Then the team assembled the dataset called "LongWriter-6k," which is a dataset that holds 6,000 written documents ranging in length from 2,000 to 32,000 words. The modified LLM was then trained using the LongWriter-6k and the team discovered that doing so increased the word length of documents, producing texts roughly 10,000 words in length.

The technology is detailed in the article, “LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs,” which appears in the journal arXiv.