4
$\begingroup$

I like the website chatpdf.com a lot. You can upload a PDF file and then discuss the textual content of the file with the file "itself". It uses ChatGPT.

I would like to program something similar. But I wonder how to use the content of long PDF files in a ChatGPT prompt, as ChatGPT only accepts 4096 tokens per conversation.

How can I reduce the number of tokens needed?

$\endgroup$

2 Answers 2

7
$\begingroup$

I didn't find the site you mention that useful - perhaps it is not using the latest GPT model? Or ChatGPT does not yet have very good ability to understand the pdf file I provided it.

Boring answer #1 ask GPT to summarize each page or section and then feed the summaries to it.

Boring answer #2 fine tune your own language model on this kind of tasks. As the good ones are still massive, it is unlikely you will get very good results for the price you can afford.

Boring answer #3 wait until some big company releases a better model with a longer context or memory capability.

More interesting, but still experimental answer Based on the research observation that very large LLMs are capable of doing Chain of Thought reasoning, you can try to teach ChatGPT to use tools. The method is described in the ReAct paper. Related keywords are "langchain" or "Language Chain". One implementation can be found here. There is a python framework called LangChain and an OpenAI Retrieval plugin.

Thus in your case a tool could be reading a particular page that ChatGPT requests. Once it requests reading a particular page, you (or your program) would input that page and so on, until it arrives to the final answer. Another tool would be to give the model access to the text search (Ctrl+F).

Example (from here):

You are working with a pandas dataframe in Python. The name of the dataframe is `df`.
You should use the tools below to answer the question posed of you.

python_repl_ast: A Python shell. Use this to execute python commands. Input should be a valid python command. When using this tool, sometimes output is abbreviated - make sure it does not look abbreviated before using it in your answer.

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [python_repl_ast]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question
```
$\endgroup$
0
$\begingroup$

A typical solution is to chunk the long text into smaller fragments, retrieve relevant pieces based on the search query, then send them via an API call.

Here's a project available which can handle PDFs, txt and doc files, as well as web pages. I believe this implementation is straightforward to follow, if you're interested.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.