I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the brand names, company names, like NLTK etc. I need some solid solution for the purpose.

Here's what I tried:

def removeNonEnglishWordsFunct(x):
    words = set(nltk.corpus.words.words())
    filteredSentence = " ".join(w for w in nltk.wordpunct_tokenize(x) \
                                if w.lower() in words or not w.isalpha())
    return filteredSentence

string = "NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Wは比較的新しくてきれいなのですが Sheraton hotelは時々 NYらしい小さくて清潔感のない部屋"

res = removeNonEnglishWordsFunct(string)
Output: testing man Apple Al Palace

Expected output: NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Sheraton hotel

To Tokenise, clean up symbols (i.e. Normalise), etc. just use one of the widely used NLP libraries, they should be able to do most of the work for you.

Examples include:

  • NTLK
  • Spacy
  • SparkNLP .. and many more. Perhaps look up some articles comparing their strengths and weaknesses on Google to decide what's best with your project.

As for the detecting English words, that might be slightly trickier, but you can find answers to this already from a bit of Googling. E.g. https://intellipaat.com/community/5638/removing-non-english-words-from-text-using-python

Might also be worth posting some code, output examples and what you're intending to do down the line (e.g. training a neural network?) so that other's can provide further help.

All the best, Kelvin


To do this, simply create a column with the language of the review and filter non-English reviews. To detect languages, I'd recommend using langdetect.

This would like something like this:

import pandas as pd

def is_english(text):
    // Add language detection code here
    return True // or False

cleaned_df = df[is_english(df["review”])]
  • $\begingroup$ langdetect unfortunately is not very precise, Fasttext models are usually better: alexott.blogspot.com/2017/10/… $\endgroup$ – Alex Ott Jun 29 '20 at 10:41
  • $\begingroup$ Interesting. Are there any benchmarks? I do like langdetect because it is a standalone solution with few dependencies and is usually quite fast since it's a bayesian model $\endgroup$ – Valentin Calomme Jun 29 '20 at 11:48
  • 1
    $\begingroup$ see the link that I sent... $\endgroup$ – Alex Ott Jun 29 '20 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.