1
$\begingroup$

What is considered short and long text in NLP?

I'm working on a dataset that contains documents from 10 to 600 words and I'm asking myself if I should treat them differently. Also, I haven't found a source which explicitly defines short and long text in NLP yet. The goal for my task is to find similar documents.

$\endgroup$
1
  • $\begingroup$ Well, it's like in general language, it depends... in your data short is close to 10 and long is close to 600, but I guess that's not the answer you expect ;) $\endgroup$
    – Erwan
    Nov 10 '20 at 19:50
1
$\begingroup$

As Erwan said in the comments, it depends. In my experience, it depends specifically on two things:

Tokenization method: The length of a document in number of tokens will vary considerably depending on how you split it up. Splitting your text into individual characters will result in a longer document than splitting it into sub-word units (e.g. WordPiece), which will still be longer than splitting on white space.

Model: Vanishing gradients aside, an RNN doesn't care how long the input text is, it will just keep chugging along. Transformers, however, are limited. BERT can realistically handle sequences of up to 512 WordPiece units, while the LongFormer claims to handle sequences of up to 32k units (given sufficient compute resources). Thus your documents of 10 - 600 tokens would be long for BERT but short for the LongFormer.

Whether you should treat documents of length 10 differently from those of length 600 is not something I can answer without knowing the details of your specific task. Intuitively, I doubt a very short document would ever be very similar to a much longer one, simply because it likely contains less content.

$\endgroup$
2
  • 1
    $\begingroup$ Thank you for your answer and sorry for the late reply. I guess that means I have to set a minimum of words which I consider as document as well as a maximum depending on my model. I hoped that there was a rule of thumb. $\endgroup$
    – jonas
    Nov 16 '20 at 17:59
  • 1
    $\begingroup$ Explicitly defining a minimum and maximum is the way to go. This is one of the reasons why many papers include tables of summary statistics of their corpora. $\endgroup$ Nov 17 '20 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.