How to create a good list of stopwords

Question

I am looking for some hints on how to curate a list of stopwords. Does someone know / can someone recommend a good method to extract stopword lists from the dataset itself for preprocessing and filtering?

The Data:

a huge amount of human text input of variable length (searchterms and whole sentences (up to 200 characters) ) over several years. The text contains a lot of spam (like machine input from bots, single words, stupid searches, product searches ... ) and only a few % of seems to be useful. I realised that sometimes (only very rarely) people search my side by asking really cool questions. These questions are so cool, that i think it is worth to have a deeper look into them to see how people search over time and what topics people have been interested in using my website.

My problem:

is that i am really struggling with the preprocessing (i.e. dropping the spam). I already tried some stopword list from the web (NLTK etc.), but these don't really help my needs regarding this dataset.

Thanks for your ideas and discussion folks!

Answer 1

One approach would be to use tf-idf score. The words which occur in most of the queries will be of little help in differentiating the good search queries from bad ones. But ones which occur very frequently (high tf or term-frequency) in only few queries (high idf or inverse document frequency) as likely to be more important in distinguishing the good queries from the bad ones.

Answer 2

It depends on your application.

When you are doing topic modelling, try the default stopwords first. When there are some words occurring prominently in many topics (note my rather vague formulation) they are good candidates for additional stopwords.

E.g., in a corpus with texts containing figures and tabular material, the words "fig", "figure", "tab", or "table" are good additional stopwords. In the result, your topics become more well-defined.

Answer 3

Using TFIDF (term frequency inverse document frequency) will solve your purpose. Get the TFIDF score for each word in your document and sort the words by their scores by which you can select the important words in your data.

Answer 4

An approach I have used to build a stopword list is to build and train a logistic regression model (due to its interpretability) on your text data. Take the absolute value of the coefficients for each token. Then, sort descending the absolute value of the coefficients of the tokens. Then, create a list of all the tokens with high coefficient absolute value that might lead to overfitting or that might meet some other criteria to be a stopword. That list is your stopwords list. You can then apply that stopword list to another set of documents of this type (kind of like a test set), to see if removing them increases the accuracy, precision, or recall of the test set model.

This strategy is effective because it takes into account the impact of tokens when building a stopword list.

Answer 5

Stopwords may be part of the solution at some point, but not the key. In any case for any major languages good lists of stop words exist, it should not be domain specific.

I also don't think that using TD-IDF alone is really correct. There could be very rare (potentially garbage) words in poor quality strings.

Instead of trying to guess which exact features are useful: I would start by creating a data set by randomly selecting some of the data and labeling them by hand (as good or bad, or on a scale from 0.0 to 1.0). Then code something up that pulls out many features (length, number of words (tokens), spam score, whether it contains URLs or botish chars, detected language, whether it has a question mark, whether it has proper capitalisation). Also don't forget to include non-linguistic features that you may have, like country of the geoIP of the user that made the query, whether the user was logged in, how old the user's account is. So at this point you will have a massive table/CSV, and a smaller one with one extra column for the label you've added.

Then train some machine learning package with those labeled examples to build a model that is accurate enough for you. Then let that model run on the rest of the data.

If you wish not to code too much, you could even just get those features into CSV form, and give them to the Google Prediction API's spreadsheet interface.