3
$\begingroup$

I have created document-term matrix using TfIdfVectorizer, but just noticed the feature contains Chinese characters. Is it possible to remove them using Python's regex?

I believe these characters are one of reason for lower prediction accuracy of my model.

Currently I use the below for pre-processing my data-

   # Pre-processing the data
    def text_preprocess( data ):
        # Changing to lower case
        data = data.lower()
        # Removing special characters
        data = re.sub("(\\d|\\W)+"," ",data)
        return data

Also, please note I used stopwords='english' in my TfidfVectorizer.

Please let me know if any information required. (New here, still learning)

$\endgroup$

1 Answer 1

6
$\begingroup$

If you want to remove non-English characters then this regex will work, by selecting characters not in a given ASCII range (0 to 122, you can adjust this since it will allow some special characters):

([^\x00-\x7A])+

So to remove those characters:

data = re.sub("([^\x00-\x7F])+"," ",data)
$\endgroup$
1
  • 1
    $\begingroup$ Perfect. Even I was thinking on same line, like excluding all non-keyboard characters. But then realised, someone might have Chinese characters on their keyboards. :) You rightly pointed at the ASCII codes. Thanks. $\endgroup$
    – ranit.b
    Mar 6, 2019 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.