Increasing SpaCy max NLP limit

Question

I'm getting this error:

[E088] Text of length 1029371 exceeds maximum of 1000000. The v2.x parser and NER models require roughly 1GB of temporary memory per 100,000 characters in the input. This means long texts may cause memory allocation errors. If you're not using the parser or NER, it's probably safe to increase the `nlp.max_length` limit. The limit is in number of characters, so you can check whether your inputs are too long by checking `len(text)`.

The weird thing is that if I reduce the amount of documents being lemmatized, it still says the length exceeds 1 million. Is there a way of increasing the limit past 1 million? The error seems to suggest there is but I'm unable to do so.

Answer 1

Try to raise the nlp.max_length parameter (as your error message suggests):

nlp = spacy.load('en_core_web_sm') 
nlp.max_length = 1500000 #or any large value, as long as you don't run out of RAM

also, when calling your spaCy pipeline, you can disable RAM-hungry intensive parts of the pipeline that are not needed for lemmatization:

doc = nlp("The sentences we'd like to do lemmatization on", disable = ['ner', 'parser'])

Finally, you should get the results you expect with the following:

print([x.lemma_ for x in doc])

Answer 2

TLDR: Dynamically set the nlp.max_length according to the length of the document. This makes it simpler while handling documents/text of unknown length.

Or you can remove some of the parts of SpaCy object pipeline which you will not need.

Let us say: txt --> text document

then set nlp.max_length = len(txt) + 100 (100 is just a cushion not necessary really)

Example: I faced the same issue, I had to loop over a directory of text files and perform NER on the text files to extract entities present in them.

for file in folder_text_files:
    with open(file, 'r', errors="ignore") as f:
         text = f.read()
         f.close()
    nlp.max_length = len(text) + 100

So doing this might help you worrying about the text size.

Answer 3

I wasn't able to figure out how to increase the maximum limit of characters but I did however just split my document in half. The problem is that SpaCy cannot process more than 1 million characters. Because I ran into this problem during the lemmatization, it doesn't matter if the document is one whole or a few parts.