I am new to NLP and have a dataset that has a bunch of (social media) messages on which I would like to try some methods like latent Dirichlet allocation (LDA). First, I need to clean the data of things like punctuation, emojis, etc. I'm not sure how to go about doing this in the most efficient and accurate manner. My code right now is this:

import pandas as pd
import re

class TopicModel():
    def __init__(self, data_path = "data.csv"):
        self.data_path = data_path
        self.data = pd.read_csv(self.data_path, low_memory=False)

    def clean_data(self):

    def remove_message_na(self):
        self.data = self.data.loc[~pd.isna(self.data['message'])]

    def remove_emojis(self):
        self.data['message'] = self.data['message'].str.encode("ascii", "ignore").str.decode("utf8")

    def remove_punctuation_and_lower(self):
        p = re.compile('''[!#?,.:";]''')
        self.data['cleaned_data'] = [p.sub("", ii).lower() for ii in self.data['message'].tolist()]

    def remove_empty_messages(self):
        self.data = self.data.loc[self.data['cleaned_data'] != ""]

    def remove_url(self):
        self.data = [re.sub(r"http\S+", "", ii) for ii in self.data['message'].tolist()]

I don't want to remove contractions, which is why I left out ' from my punctuation list, but I think optimally, the contractions would be reformatted as two separate words. I also wonder about the other punctuation marks when dealing with social media data, e.g., #. I know this question is a bit general, but I'm wondering if there is a good python library for performing the kind of data-cleaning operations that I want, prior to perform topic analysis, sentiment analysis, etc. I'd also like to know which libraries can efficiently perform these data-cleaning operations on a pandas data frame.

  • $\begingroup$ Commenting on how you ask the questions. You need to make your questions more clear by grouping them up or specifically putting them in bullet points. I find it confusing. $\endgroup$ – Fatemeh Rahimi Jan 9 at 5:22

I summarize your questions and then try to answer under each bullet point:

  • How to remove punctuation marks (e.g. # for hashtags which is used in social media)

The first goto is a regular expression that is used in data preprocessing very frequently. But if you are looking for all your punctuation to be removed from the text you can use one of these two approaches:

import string
sentence = "hi; how*, @are^ you? wow!!!"
sentence = sentence.translate(sentence.maketrans('', '', string.punctuation))

output: 'hi how are you wow'

or use a regular expression, example:

import re
s = "string. With. Punctuation? 3.2"
s = re.sub(r'[^\w\s]','',s)

output: 'string With Punctuation 32'

  • What are the possible python library for data cleaning?

generally, NLTK and SciPy are very handly. But for specific purposes, other libraries also exist. For example library contractions, inflect.

  • Which libraries are efficient in data cleaning when using pandas dataframes?

You can apply any function from any library to your pandas dataframe using the apply method. Here is an example:

mport pandas as pd 
s = pd.read_csv("stock.csv", squeeze = True) 
# adding 5 to each value 
new = s.apply(lambda num : num + 5) 

code source

  • $\begingroup$ Thanks for this response. I'm trying to understand best practices for handling contractions. First question: Should contractions be eliminated? The use of string.punctuation in your suggested code would lead to confusion over words like I'm or they're, transforming them to Im and theyre. I'm interested in understanding best practice solutions to this. Second question: How should we treat hyphenated words? Is there a way to keep hyphenated words, but also getting rid of the hyphen when it's used otherwise, e.g., a minus sign? $\endgroup$ – CopyOfA Jan 11 at 21:27
  • $\begingroup$ To answer the first question: I personally like to make the text as clean as possible so that it is clear for my model, so I suggest replacing the contractions with the full version(i.e. I'm to I am). But if I'm using a pre-trained model that is trained on a huge amount of text, then I don't worry about it because these models have already seen these words many times and they can understand the context of sentences perfectly. $\endgroup$ – Fatemeh Rahimi Jan 17 at 21:18
  • $\begingroup$ Regarding the second question, It again depends on your model. If it's an ML model that is counting the occurrence of the words, so removing the hyphens makes more sense, otherwise, if it's a pre-trained language model, then I suggest just passing it to the model tokenizer as it is. $\endgroup$ – Fatemeh Rahimi Jan 17 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.