0
$\begingroup$

I'm wanting to count how often Chinese is used, I've tried langdetect but the issue is the values are a mix of numbers, text and even URLs which langdetect doesn't seem to support

import pandas as pd
from langdetect import detect

df = pd.read_csv("annotation_sample.csv", nrows=10)
df = df.drop(['id', 'profile_id', 'created', 'document_id'], axis=1)
df = df[pd.notnull(df['txt'])]
df

df.txt.apply(lambda x: detect(x))

Above works (although I've not figured out how to count a specific language) although it only works as its limited to 10 rows, when using the entire 10,000 it throws the 'No features in text' error

Pic of the data:

pic of data

What I need help with is; How do I avoid running this on rows that contain just numbers, urls etc? Is there a way to skip error'd ones and continue?

Secondly, how do I then count how many have come back as a specific language?

$\endgroup$

1 Answer 1

1
$\begingroup$

That should be feasible using some more python and pandas features: - errors can be caught when you wrap the detect in your own function and return None (or what you prefer) instead of a language code - results of the detect can be added to the df and counted, filtered, you name it.

def mydetect(txt):
    try:
        return detect(txt)
    except:
        return None

Pandas allows you to add the results to a second column of the dataframe, but I do not have the command handy at the moment, please check a tutorial. Then you can filter the dataset or count occurences of a language.

You can also limit the except clause to certain errors if you think that would make more sense.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.