The domain here is essentially topic classification, so not necessarily a problem where stop-words have an impact on the analysis (as opposed to, say, sentiment analysis where structure can affect meaning).

With respect to the positional encoding mechanism in transformer language models, when using a pretrained LM is stop-word removal as a preprocessing step actively harmful if the LM was trained on a corpus where they were left in? I'm still working on fully understanding the mechanism but I feel like removing stop-words would affect which wavelength is used to construct the context between any given pair of words with stop words between them, which in turn would impact the encoding.

Or, would this not matter because the regression when trained figures it out from consistently processed input? I feel like it should matter but haven't been able to find anything on the topic.

  • $\begingroup$ The more I think about it, the more it seems like it's a bad idea. If it wasn't, why bother working on all of these different encoding mechanisms? Nobody I've asked has had any idea, though. $\endgroup$
    – Oso
    Dec 8 '20 at 16:42

Very interesting question.

  • Easy, but probably lazy answer

When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't remove them, and if it doesn't matter, it doesn't hurt to keep them in. Obviously, if you can, try with or without stopwords, and see what works best for your problem.

  • Longer answer

You actually have two ways to "remove" your stopwords. Either you remove them from the input sequence altogether. Or, you could replace them with a mask token (i.e. <#UNKNOWN> or <#MASK>).

In the latter case, the transformer will implicitly guess what these masks are and you will have achieved the original goal of stopwords removal: make sure they don't affect the predictive outcome. Indeed, take the sentences:

"I like basketball with an audience" & "I like basketball without an audience"

These two sentences are both about basketball and you wouldn't want with/without to make your model think these sentences are about different topics. By masking with and without, you both "remove" the stopwords, and you don't potentially mess with the fact that the pre-trained model didn't use data without stopwords.

Now, what would happen if you fed an "incomplete" sentence to a transformer. The positional encodings would retain the notion that certain words are before or after each other, which is what you want. But if you remove stopwords, some words may appear "too close" to each other. But does that matter? I do not think so.

If the vector your transformer outputs for "word1 <#Mask> word2" is widely different than the one for "word1 word2", it must mean that the masked token is crucial to the overall meaning of the sentence, which would also suggest that it shouldn't be a stopword to begin with.

  • Final answer

I would suggest masking the stopwords instead of removing them. But, if performance is so critical that you need to feed smaller sequences, I think that you will be alright.

  • 1
    $\begingroup$ I hadn't considered masking them, that's a brilliant compromise that sidesteps the problem entirely. I'll accept if nobody is able to provide sources for the technical bit, thank you very much. $\endgroup$
    – Oso
    Jan 5 at 18:28

Removing stop words or keeping them is an empirical question. The effect will vary based on corpus and task. In fact, the definition of stop words depends on the corpus and the task.

One approach would be to benchmark the effect of stop words with cross validation for the specific scenario.

  • $\begingroup$ Thanks for your response, but the question is more about the specific mechanics of positional encoding in transformer language models and if using a shorter wavelength than actual due to stop word removal has an adverse impact on the encoded representation of the sequence. I think it should be answerable. $\endgroup$
    – Oso
    Jan 5 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.