9
$\begingroup$

I'm getting this error:

[E088] Text of length 1029371 exceeds maximum of 1000000. The v2.x parser and NER models require roughly 1GB of temporary memory per 100,000 characters in the input. This means long texts may cause memory allocation errors. If you're not using the parser or NER, it's probably safe to increase the `nlp.max_length` limit. The limit is in number of characters, so you can check whether your inputs are too long by checking `len(text)`.

The weird thing is that if I reduce the amount of documents being lemmatized, it still says the length exceeds 1 million. Is there a way of increasing the limit past 1 million? The error seems to suggest there is but I'm unable to do so.

$\endgroup$
3
  • 1
    $\begingroup$ What code exactly are you running when you get that error? Please include a sapmle in your post. $\endgroup$
    – n1k31t4
    Sep 25 '18 at 0:30
  • $\begingroup$ Facing the same issue. Would be nice if spacy leaves it to the user how many words his/her infrastructure can process. $\endgroup$
    – padmalcom
    Jan 2 '19 at 13:01
  • $\begingroup$ See my answer. I spent many hours trying to troubleshoot this and figured that it was just easier to split the document into smaller pieces. Initially, I thought it had to do with the amount of RAM I was running.. But I think its a character limit on the library $\endgroup$
    – D500
    Jan 3 '19 at 14:03
10
$\begingroup$

Try to raise the nlp.max_length parameter (as your error message suggests):

nlp = spacy.load('en_core_web_sm') 
nlp.max_length = 1500000 #or any large value, as long as you don't run out of RAM

also, when calling your spaCy pipeline, you can disable RAM-hungry intensive parts of the pipeline that are not needed for lemmatization:

doc = nlp("The sentences we'd like to do lemmatization on", disable = ['ner', 'parser'])

Finally you should get the results you expect with:

print([x.lemma_ for x in doc])
$\endgroup$
0
$\begingroup$

I wasn't able to figure out how to increase the maximum limit of characters but I did however just split my document in half. The problem is that SpaCy cannot process more than 1 million characters. Because I ran into this problem during the lemmatization, it doesn't matter if the document is one whole or a few parts.

$\endgroup$
0
$\begingroup$

TLDR: Dynamically set the nlp.max_length according to the length of the document. This makes it simpler while handling documents/text of unknown length.

Or you can remove some of the parts of SpaCy object pipeline which you will not need.

Let us say: txt --> text document

then set nlp.max_length = len(txt) + 100 (100 is just a cushion not necessary really)

Example: I faced the same issue, I had to loop over a directory of text files and perform NER on the text files to extract entities present in them.

for file in folder_text_files:
    with open(file, 'r', errors="ignore") as f:
         text = f.read()
         f.close()
    nlp.max_length = len(text) + 100

So doing this might help you worrying about the text size.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.