I'm getting this error:

[E088] Text of length 1029371 exceeds maximum of 1000000. The v2.x parser and NER models require roughly 1GB of temporary memory per 100,000 characters in the input. This means long texts may cause memory allocation errors. If you're not using the parser or NER, it's probably safe to increase the `nlp.max_length` limit. The limit is in number of characters, so you can check whether your inputs are too long by checking `len(text)`.

The weird thing is that if I reduce the amount of documents being lemmatized, it still says the length exceeds 1 million. Is there a way of increasing the limit past 1 million? The error seems to suggest there is but I'm unable to do so.

  • 1
    $\begingroup$ What code exactly are you running when you get that error? Please include a sapmle in your post. $\endgroup$
    – n1k31t4
    Sep 25 '18 at 0:30
  • $\begingroup$ Facing the same issue. Would be nice if spacy leaves it to the user how many words his/her infrastructure can process. $\endgroup$
    – padmalcom
    Jan 2 '19 at 13:01
  • $\begingroup$ See my answer. I spent many hours trying to troubleshoot this and figured that it was just easier to split the document into smaller pieces. Initially, I thought it had to do with the amount of RAM I was running.. But I think its a character limit on the library $\endgroup$
    – D500
    Jan 3 '19 at 14:03

Try to raise the nlp.max_length parameter (as your error message suggests):

nlp = spacy.load('en_core_web_sm') 
nlp.max_length = 1500000 #or any large value, as long as you don't run out of RAM

also, when calling your spaCy pipeline, you can disable RAM-hungry intensive parts of the pipeline that are not needed for lemmatization:

doc = nlp("The sentences we'd like to do lemmatization on", disable = ['ner', 'parser'])

Finally you should get the results you expect with:

print([x.lemma_ for x in doc])

I wasn't able to figure out how to increase the maximum limit of characters but I did however just split my document in half. The problem is that SpaCy cannot process more than 1 million characters. Because I ran into this problem during the lemmatization, it doesn't matter if the document is one whole or a few parts.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.