# Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the brand names, company names, like NLTK etc. I need some solid solution for the purpose.

Here's what I tried:

def removeNonEnglishWordsFunct(x):
words = set(nltk.corpus.words.words())
filteredSentence = " ".join(w for w in nltk.wordpunct_tokenize(x) \
if w.lower() in words or not w.isalpha())
return filteredSentence

string = "NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Ｗは比較的新しくてきれいなのですが Ｓｈｅｒａｔｏｎ hotelは時々 ＮＹらしい小さくて清潔感のない部屋"

res = removeNonEnglishWordsFunct(string)
Output: testing man Apple Al Palace

Expected output: NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Ｓｈｅｒａｔｏｎ hotel
$$$$


## 2 Answers

To Tokenise, clean up symbols (i.e. Normalise), etc. just use one of the widely used NLP libraries, they should be able to do most of the work for you.

Examples include:

• NTLK
• Spacy
• SparkNLP .. and many more. Perhaps look up some articles comparing their strengths and weaknesses on Google to decide what's best with your project.

As for the detecting English words, that might be slightly trickier, but you can find answers to this already from a bit of Googling. E.g. https://intellipaat.com/community/5638/removing-non-english-words-from-text-using-python

Might also be worth posting some code, output examples and what you're intending to do down the line (e.g. training a neural network?) so that other's can provide further help.

All the best, Kelvin

To do this, simply create a column with the language of the review and filter non-English reviews. To detect languages, I'd recommend using langdetect.

This would like something like this:

import pandas as pd

def is_english(text):
// Add language detection code here
return True // or False

cleaned_df = df[is_english(df["review”])]
`
• langdetect unfortunately is not very precise, Fasttext models are usually better: alexott.blogspot.com/2017/10/… – Alex Ott Jun 29 '20 at 10:41
• Interesting. Are there any benchmarks? I do like langdetect because it is a standalone solution with few dependencies and is usually quite fast since it's a bayesian model – Valentin Calomme Jun 29 '20 at 11:48
• see the link that I sent... – Alex Ott Jun 29 '20 at 11:51