0
$\begingroup$

My question is about when to balance training data for sentiment analysis. Upon evaluating my training dataset, which has 3 labels (good, bad, neutral), I noticed there were twice as many neutral labels as the other 2 combined so I used a function to drop neutral labels randomly. However, I wasn't sure if I should do this before or after creating the vocab2index mappings. To explain, I am numericising my text data by creating a vocabulary of words in the training data and linking them to numbers using enumerate. I think use that dictionary of vocab:index values to numericise the training data. I also use that same dictionary to numericise the testing data, dropping any words that do not exist in the dictionary. When I took a class on this, they had balanced the training data AFTER creating the vocab2index dictionary. However, when I thought about this in my own implementation, it did not make sense. What if some words from the original vocabulary are gone completely, then we aren't training the machine learning classifier on those words but they would not be dropping from the testing data either (since words are dropping from X_test based on whether they are in the vocab2index dictionary). So should I be balancing the data BEFORE creating the vocab2index dictionary? I linked the code to create X_train and X_test below in case it helps. Thanks!

def create_X_train(training_data='Sentences_75Agree_csv.csv'):
    data_csv = pd.read_csv(filepath_or_buffer=training_data, sep='.@', header=None, names=['sentence','sentiment'], engine='python')
    list_data = []
    for index, row in data_csv.iterrows():
        dictionary_data = {}
        dictionary_data['message_body'] = row['sentence']
        if row['sentiment'] == 'positive':
             dictionary_data['sentiment'] = 2
        elif row['sentiment'] == 'negative':
             dictionary_data['sentiment'] = 0
        else:
             dictionary_data['sentiment'] = 1 # For neutral sentiment
        list_data.append(dictionary_data)
    dictionary_data = {}
    dictionary_data['data'] = list_data
    messages = [sentence['message_body'] for sentence in dictionary_data['data']]
    sentiments = [sentence['sentiment'] for sentence in dictionary_data['data']]

    tokenized = [preprocess(sentence) for sentence in messages]
    bow = Counter([word for sentence in tokenized for word in sentence]) 
    freqs = {key: value/len(tokenized) for key, value in bow.items()} #keys are the words in the vocab, values are the count of those words
    # Removing 5 most common words from data
    high_cutoff = 5
    K_most_common = [x[0] for x in bow.most_common(high_cutoff)] 
    filtered_words = [word for word in freqs if word not in K_most_common]
    # Create vocab2index dictionary:
    vocab = {word: i for i, word in enumerate(filtered_words, 1)}
    id2vocab = {i: word for word, i in vocab.items()}
    filtered = [[word for word in sentence if word in vocab] for sentence in tokenized] 

    # Balancing training data due to large number of neutral sentences
    balanced = {'messages': [], 'sentiments':[]}
    n_neutral = sum(1 for each in sentiments if each == 1)
    N_examples = len(sentiments)
    # print(n_neutral/N_examples)
    keep_prob = (N_examples - n_neutral)/2/n_neutral
    # print(keep_prob)
    for idx, sentiment in enumerate(sentiments):
        message = filtered[idx]
        if len(message) == 0:
            # skip this sentence because it has length 0
            continue
        elif sentiment != 1 or random.random() < keep_prob:
            balanced['messages'].append(message)
            balanced['sentiments'].append(sentiment)

    token_ids = [[vocab[word] for word in message] for message in balanced['messages']]
    sentiments_balanced = balanced['sentiments']
    # Unit test:
    unique, counts = np.unique(sentiments_balanced, return_counts=True)
    print(np.asarray((unique, counts)).T)
    print(np.mean(sentiments_balanced))
    ##################
    # Left padding and truncating to the same length 
    X_train = token_ids
    for i, sentence in enumerate(X_train):
        if len(sentence) <=30:
            X_train[i] = ((30-len(sentence)) * [0] + sentence)
        elif len(sentence) > 30:
            X_train[i] = sentence[:30]
    return vocab, X_train, sentiments_balanced
def create_X_test(test_sentences, vocab):
    tokenized = [preprocess(sentence) for sentence in test_sentences]
    filtered = [[word for word in sentence if word in vocab] for sentence in tokenized] # X_test filtered to only words in training vocab
    # Alternate method with functional programming:
    # filtered = [list(filter(lambda a: a in vocab, sentence)) for sentence in tokenized]
    token_ids = [[vocab[word] for word in sentence] for sentence in filtered] # Numericise data
    # Remove short sentences in X_test
    token_ids_filtered = [sentence for sentence in token_ids if len(sentence)>10]
    X_test = token_ids_filtered
    for i, sentence in enumerate(X_test):
        if len(sentence) <=30:
            X_test[i] = ((30-len(sentence)) * [0] + sentence)
        elif len(sentence) > 30:
            X_test[i] = sentence[:30]
    return X_test
$\endgroup$
0
$\begingroup$

I will try to answer with best of my capabilities and knowledge acquired .

1) Do not drop Good data just to balance it with lesser class

2) You can use imbalanced-learn and Smote based Python Libraries to work on Imbalanced data but as its a text data, your results might be not so fruitful as you expect. Again it also depends on the domain of the text language.

3) Use Ensemble/Voting mechanism/Boosting techniques to improve your model.

I hope this pointers help you out.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.