I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the brand names, company names, like NLTK etc. I need some solid solution for the purpose.

Here's what I tried:

def removeNonEnglishWordsFunct(x):
    words = set(nltk.corpus.words.words())
    filteredSentence = " ".join(w for w in nltk.wordpunct_tokenize(x) \
                                if w.lower() in words or not w.isalpha())
    return filteredSentence

string = "NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Wは比較的新しくてきれいなのですが Sheraton hotelは時々 NYらしい小さくて清潔感のない部屋"

res = removeNonEnglishWordsFunct(string)
Output: testing man Apple Al Palace

Expected output: NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Sheraton hotel

3 Answers 3


To Tokenise, clean up symbols (i.e. Normalise), etc. just use one of the widely used NLP libraries, they should be able to do most of the work for you.

Examples include:

  • NTLK
  • Spacy
  • SparkNLP .. and many more. Perhaps look up some articles comparing their strengths and weaknesses on Google to decide what's best with your project.

As for the detecting English words, that might be slightly trickier, but you can find answers to this already from a bit of Googling. E.g. https://intellipaat.com/community/5638/removing-non-english-words-from-text-using-python

Might also be worth posting some code, output examples and what you're intending to do down the line (e.g. training a neural network?) so that other's can provide further help.

All the best, Kelvin


If we are looking to remove Non-English words in a column, we can simply do it using regular expressions.

Here is what I tried while cleaning tweets for sentiment analysis-

new_string=re.sub('[^a-zA-Z0-9]',' ',string)

cleaned_string=re.sub('\s+',' ',new_string)

  • $\begingroup$ your approach removes ONLY the characters not words, for example: sent = "Löfven: Så jämnt att man inte kan garantera något" new_string=re.sub('[^a-zA-Z0-9]',' ',sent) cleaned_string=re.sub('\s+',' ',new_string) cleaned_string which returns: L fven S j mnt att man inte kan garantera n got $\endgroup$
    – doplano
    Sep 11, 2022 at 18:46

To do this, simply create a column with the language of the review and filter non-English reviews. To detect languages, I'd recommend using langdetect.

This would like something like this:

import pandas as pd

def is_english(text):
    // Add language detection code here
    return True // or False

cleaned_df = df[is_english(df["review”])]
  • $\begingroup$ langdetect unfortunately is not very precise, Fasttext models are usually better: alexott.blogspot.com/2017/10/… $\endgroup$
    – Alex Ott
    Jun 29, 2020 at 10:41
  • $\begingroup$ Interesting. Are there any benchmarks? I do like langdetect because it is a standalone solution with few dependencies and is usually quite fast since it's a bayesian model $\endgroup$ Jun 29, 2020 at 11:48
  • 1
    $\begingroup$ see the link that I sent... $\endgroup$
    – Alex Ott
    Jun 29, 2020 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.