10
$\begingroup$

What is the best way to summarize a long text that exceeds 4096 token limit (like a podcast transcript for example)? As I understand I need to split the text into chunks to summarize, and then concatenate the results and summarize those. Is there already a popular open-source script to do that?

Do I understand correctly that GPT-3 is the best model to do that? I've seen some articles about extractive summarization using BERT but the results were pretty low quality.

$\endgroup$
3
  • $\begingroup$ I think GPT-3 can only be accessed through an API currently. $\endgroup$
    – anon
    Commented Jan 12, 2023 at 9:49
  • 1
    $\begingroup$ @lynx and beta.openai.com/playground $\endgroup$ Commented Jan 17, 2023 at 4:54
  • 1
    $\begingroup$ @lynx both API and Playground don't support texts larger than 4096 tokens $\endgroup$
    – Poma
    Commented Jan 20, 2023 at 9:56

1 Answer 1

5
$\begingroup$

Is there already a popular open-source script to do that?

The Python library GPT Index (MIT license) can summarize a large document or collection of documents with GPT-3.

From the documentation:

index = GPTTreeIndex(documents)
response = index.query("<summarization_query>", mode="summarize")

The “default” mode for a tree-based query is traversing from the top of the graph down to leaf nodes. For summarization purposes we will want to use mode="summarize".

 A summarization query could look like one of the following:

  • “What is a summary of this collection of text?”
  • “Give me a summary of person X’s experience with the company.”

The documentation includes a notebook with complete examples: https://github.com/jerryjliu/gpt_index/blob/main/examples/paul_graham_essay/TestEssay.ipynb


Another Python library: https://github.com/hwchase17/langchain (MIT license). From the documentation:

from langchain.chains.summarize import load_summarize_chain
chain = load_summarize_chain(llm, chain_type="map_reduce")
chain.run(docs)

FYI {1,2} are two great papers looking at GPT-3 performance for summarization, but they only looked at short texts.

Update 2023-02-23: the next version of GPT may allow 32k tokens:

enter image description here


Update 2023-11-15: Interesting leaderboard for summarization of relatively short documents: https://github.com/vectara/hallucination-leaderboard

enter image description here


References:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.