I am running a hate speech classifier published by Davidson et al.

The principle is simple, the classifier takes as an input an annotated ('hateful', 'offensive', 'neither') dataset of tweets. It then calculates several features (e.g., TF-IDF, part-of-speech, sentiment, etc.) and uses logistic regression to make predictions.

The authors have shared an iPython version here which I have rewritten as a standard Python script (see below). Their data, in case anyone wants to test the code is here.

from warnings import filterwarnings
filterwarnings("ignore", category=UserWarning)
filterwarnings("ignore", category=FutureWarning)
import datetime
import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
import nltk
from nltk.stem.porter import *
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer as VS
from textstat.textstat import *
from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import SelectFromModel
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedKFold, GridSearchCV
from sklearn.pipeline import Pipeline
import matplotlib.pyplot as plt

INPUT_PATH = 'DavidsonDataset.csv'

stopwords = nltk.corpus.stopwords.words("english")

other_exclusions = ["#ff", "ff", "rt"]

stemmer = PorterStemmer()
sentiment_analyzer = VS()

def preprocess(text_string):

    space_pattern = '\s+'
    giant_url_regex = ('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|'
    mention_regex = '@[\w\-]+'
    parsed_text = re.sub(space_pattern, ' ', text_string)
    parsed_text = re.sub(giant_url_regex, '', parsed_text)
    parsed_text = re.sub(mention_regex, '', parsed_text)
    return parsed_text

def tokenize(tweet):

    tweet = " ".join(re.split("[^a-zA-Z]*", tweet.lower())).strip()
    tokens = [stemmer.stem(t) for t in tweet.split()]
    return tokens

def basic_tokenize(tweet):

    tweet = " ".join(re.split("[^a-zA-Z.,!?]*", tweet.lower())).strip()
    return tweet.split()

def count_twitter_objs(text_string):

    space_pattern = '\s+'
    giant_url_regex = ('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|'
    mention_regex = '@[\w\-]+'
    hashtag_regex = '#[\w\-]+'
    parsed_text = re.sub(space_pattern, ' ', text_string)
    parsed_text = re.sub(giant_url_regex, 'URLHERE', parsed_text)
    parsed_text = re.sub(mention_regex, 'MENTIONHERE', parsed_text)
    parsed_text = re.sub(hashtag_regex, 'HASHTAGHERE', parsed_text)
    return (parsed_text.count('URLHERE'), parsed_text.count('MENTIONHERE'), parsed_text.count('HASHTAGHERE'))

def other_features(tweet):

    sentiment = sentiment_analyzer.polarity_scores(tweet)

    words = preprocess(tweet)  # Get text only

    syllables = textstat.syllable_count(words)
    num_chars = sum(len(w) for w in words)
    num_chars_total = len(tweet)
    num_terms = len(tweet.split())
    num_words = len(words.split())
    avg_syl = round(float((syllables + 0.001)) / float(num_words + 0.001), 4)
    num_unique_terms = len(set(words.split()))

    # Modified FK grade, where avg words per sentence is just num words/1
    FKRA = round(float(0.39 * float(num_words) / 1.0) + float(11.8 * avg_syl) - 15.59, 1)
    # Modified FRE score, where sentence fixed to 1
    FRE = round(206.835 - 1.015 * (float(num_words) / 1.0) - (84.6 * float(avg_syl)), 2)

    twitter_objs = count_twitter_objs(tweet)
    retweet = 0
    if "rt" in words:
        retweet = 1
    features = [FKRA, FRE, syllables, avg_syl, num_chars, num_chars_total, num_terms, num_words,
                num_unique_terms, sentiment['neg'], sentiment['pos'], sentiment['neu'], sentiment['compound'],
                twitter_objs[2], twitter_objs[1],
                twitter_objs[0], retweet]
    return features

def get_feature_array(tweets):
    feats = []
    for t in tweets:
    return np.array(feats)

vectorizer = TfidfVectorizer(
    ngram_range=(1, 3),

def main_function():

    df = pd.read_csv(INPUT_PATH)

    tweets = df.text   # Get tweets

    # Construct tfidf matrix and get relevant scores
    print("Contructing TF-IDF matrix and getting relevant scores...")
    tfidf = vectorizer.fit_transform(tweets).toarray()
    vocab = {v: i for i, v in enumerate(vectorizer.get_feature_names())}
    idf_vals = vectorizer.idf_
    idf_dict = {i: idf_vals[i] for i in vocab.values()}  # keys are indices; values are IDF scores

    # Get POS tags for tweets and save as a string
    print("Getting POS tags and saving them as a string...")
    tweet_tags = []
    for t in tweets:
        tokens = basic_tokenize(preprocess(t))
        tags = nltk.pos_tag(tokens)
        tag_list = [x[1] for x in tags]
        tag_str = " ".join(tag_list)

    # We can use the TFIDF vectorizer to get a token matrix for the POS tags
    pos_vectorizer = TfidfVectorizer(
        ngram_range=(1, 3),

    # Construct POS TF matrix and get vocab dict
    print("Constructing POS TF matrix...")
    pos = pos_vectorizer.fit_transform(pd.Series(tweet_tags)).toarray()
    pos_vocab = {v: i for i, v in enumerate(pos_vectorizer.get_feature_names())}

    other_features_names = ["FKRA", "FRE", "num_syllables", "avg_syl_per_word", "num_chars", "num_chars_total", "num_terms", "num_words", "num_unique_words", "vader neg", "vader pos", "vader neu", "vader compound", "num_hashtags", "num_mentions", "num_urls", "is_retweet"]

    print("Generating features...")
    feats = get_feature_array(tweets)

    # Now join them all up
    M = np.concatenate([tfidf, pos, feats], axis=1)

    print("Feature table shape: ")

    # Finally get a list of variable names
    variables = [''] * len(vocab)
    for k, v in vocab.items():
        variables[v] = k

    pos_variables = [''] * len(pos_vocab)
    for k, v in pos_vocab.items():
        pos_variables[v] = k

    feature_names = variables + pos_variables + other_features_names

    print("\nRunning the model...")

    X = pd.DataFrame(M)
    y = df['label'].astype(int)

    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42, test_size=0.1)

    pipe = Pipeline([('select', SelectFromModel(LogisticRegression(class_weight='balanced', penalty="l1", C=0.01))),
                     ('model', LogisticRegression(class_weight='balanced', penalty='l2'))])

    param_grid = [{}]  # Optionally add parameters here

    print("The best model is selected using a GridSearch with 5-fold CV.")

    grid_search = GridSearchCV(pipe,
                               cv=StratifiedKFold(n_splits=5, random_state=42).split(X_train, y_train),

    model = grid_search.fit(X_train, y_train)
    y_preds = model.predict(X_test)

if __name__ == "__main__":

    print('\nProcess started...\n')

    # Start timer
    start = datetime.datetime.now()

    # Run awesome code

    # End timer
    end = datetime.datetime.now()

    # Print results
    print("\nProcess finished")
    print("Total time: " + str(end - start))

The classifier works and I can produce a classification report.

The problem is that now I want to use this model to make a simple prediction. For example, I want to feed model a tweet and learn whether it's 'hateful', 'offensive', or 'neither'.

If I run the code below:

print(model.predict(["I don't like you."]))

I receive the following error:

ValueError: Expected 2D array, got 1D array instead:
array=["I don't like you."].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

There are several similar questions on StackOverflow where the answer is simply to reshape the table (as the error suggests). However, this does not work. If I run:

print(model.predict(np.array(["I don't like you."]).reshape(-1, 1)))


print(model.predict(np.array(["I don't like you."]).reshape(1, -1)))

I get the following error:

ValueError: X has a different shape than during fitting.

My question has two parts:

  1. How can I fix this? How can I use the model to make a single prediction?
  2. Is this a feature or a bug? As far as I understand this error, sklearn wants an input that has been "fitted" to have the same dimentions of the training set. Doesn't this beat the purpose of training in the first place? The goal is to instantly be able to make a prediction. It is obvious that I lack some key insight on this matter so I would be grateful if someone explained to me what I have understood wrong.

Clarification: There are several questions regarding the Reshape your data error. However, the suggested solution (i.e., simply reshape as instructed) does not work for me as shown above. More importantly, I am interested in understanding why this behavior is normal, something that is not discussed in the similar questions.


You cannot do that directly because your training data is not a text but set of features extracted from the text.You need to convert the text to list of features and then try to predict it

| improve this answer | |
  • $\begingroup$ Thanks, but how do I convert the text to a list of features? And in regards to the second part of the questions, doesn't this beat the purpose of a pretrained model? $\endgroup$ – Aventinus Feb 12 at 17:36
  • $\begingroup$ M = np.concatenate([tfidf, pos, feats], axis=1).Here is what you are exactly doing.You are converting text to features.You are giving X as an input and y as an output.So,your classifier is trained on features instead of the text. $\endgroup$ – Sri Test Feb 12 at 18:01

There is not a reshape problem. You need to transform your text in a set of features, say, vectorize it in the same way you created your dataset, in this case using TF-IDF. Just prepare a query vector applying the same TF-IDF and will work.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.