I am trying to train and evaluate a hate detection model using the HuggingFace Transformers library and this dataset. Model performance is secondary, just trying to get it going. I have preprocessed the data and tokenised it as shown below:

import pandas as pd
import numpy as np
from numpy.random import RandomState
import re
import preprocessor as p
from transformers import AutoTokenizer

# Loading raw data
original_data = pd.read_csv('../data/data.csv')

# Make a random test and train split
rng = RandomState()
train = original_data.sample(frac=0.7, random_state=rng)
test = original_data.loc[~df.index.isin(train.index)]

# Preprocessing: remove special characters using RegEx
REPLACE_NO_SPACE = re.compile("(\.)|(\;)|(\:)|(\!)|(\')|(\?)|(\,)|(\")|(\|)|(\()|(\))|(\[)|(\])|(\%)|(\\\$)|(\>)|(\<)|(\{)|(\})")
REPLACE_WITH_SPACE = re.compile("(<br\s/><br\s/?)|(-)|(/)|(:).")

# Custum function to clean the datasets
def clean_tweets(df):
  tempArr = []
  for line in df:
    # send to tweet_processor
    tmpL = p.clean(line)
    # remove puctuation
    tmpL = REPLACE_NO_SPACE.sub("", tmpL.lower()) # convert all tweets to lower cases
    tmpL = REPLACE_WITH_SPACE.sub(" ", tmpL)
  return tempArr

# clean training data
train_tweet = clean_tweets(train["tweet"])
train_tweet = pd.DataFrame(train_tweet)
# append cleaned tweets to the testing data
train["clean_tweet"] = train_tweet

# clean the test data 
test_tweet = clean_tweets(test["tweet"])
test_tweet = pd.DataFrame(test_tweet)
# append cleaned tweets to the training data
test["clean_tweet"] = test_tweet

# Tokenisation so the inputs are ready for the model
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

The above code generates a table for the test data as: enter image description here

As I understand, the next part should be the model training part and extracting the % of hate tweets. Any suggestions on implementation?


1 Answer 1


Check this article: https://medium.com/geekculture/simple-chatbot-using-bert-and-pytorch-part-1-2735643e0baa

Model training with explanation is given


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.