I'm wanting to count how often Chinese is used, I've tried langdetect but the issue is the values are a mix of numbers, text and even URLs which langdetect doesn't seem to support

import pandas as pd
from langdetect import detect

df = pd.read_csv("annotation_sample.csv", nrows=10)
df = df.drop(['id', 'profile_id', 'created', 'document_id'], axis=1)
df = df[pd.notnull(df['txt'])]

df.txt.apply(lambda x: detect(x))

Above works (although I've not figured out how to count a specific language) although it only works as its limited to 10 rows, when using the entire 10,000 it throws the 'No features in text' error

Pic of the data:

pic of data

What I need help with is; How do I avoid running this on rows that contain just numbers, urls etc? Is there a way to skip error'd ones and continue?

Secondly, how do I then count how many have come back as a specific language?


1 Answer 1


That should be feasible using some more python and pandas features: - errors can be caught when you wrap the detect in your own function and return None (or what you prefer) instead of a language code - results of the detect can be added to the df and counted, filtered, you name it.

def mydetect(txt):
        return detect(txt)
        return None

Pandas allows you to add the results to a second column of the dataframe, but I do not have the command handy at the moment, please check a tutorial. Then you can filter the dataset or count occurences of a language.

You can also limit the except clause to certain errors if you think that would make more sense.


Not the answer you're looking for? Browse other questions tagged or ask your own question.