Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
What is considered short and long text in NLP?
I'm working on a dataset that contains documents from 10 to 600 words and I'm asking myself if I should treat them differently. Also, I haven't found a source which explicitly defines short and long text in NLP yet. The goal for my task is to find similar documents.
As Erwan said in the comments, it depends. In my experience, it depends specifically on two things:
Tokenization method: The length of a document in number of tokens will vary considerably depending on how you split it up. Splitting your text into individual characters will result in a longer document than splitting it into sub-word units (e.g. WordPiece), which will still be longer than splitting on white space.
Model: Vanishing gradients aside, an RNN doesn't care how long the input text is, it will just keep chugging along. Transformers, however, are limited. BERT can realistically handle sequences of up to 512 WordPiece units, while the LongFormer claims to handle sequences of up to 32k units (given sufficient compute resources). Thus your documents of 10 - 600 tokens would be long for BERT but short for the LongFormer.
Whether you should treat documents of length 10 differently from those of length 600 is not something I can answer without knowing the details of your specific task. Intuitively, I doubt a very short document would ever be very similar to a much longer one, simply because it likely contains less content.
