0
$\begingroup$

Preface

I have an annotated text dataset on hate speech. Simply put, the dataset consists of a column called text which includes a piece of text, and a column called label which can be either 0 (non-hateful) or 1 (hateful). Let's call this dataset Dataset A.

I am using this dataset to train a very simple classifier using Logistic Regression. Code:

dataset_1_df = pd.read_csv(d_1_path, sep='\t')

text = dataset_1_df.text   
X = text
y = dataset_1_df['label'].astype(int)

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42, test_size=0.1)

clf = Pipeline([
    ('vect', CountVectorizer(max_features=10000, ngram_range=(1, 2))),
    ('tfidf', TfidfTransformer(norm='l2')),
    ('clf', LogisticRegression()),
])

clf = clf.fit(X_train, y_train)
y_preds = clf.predict(X_test)

This classifier works fine. I can produce a classification report like this:

report = classification_report(y_test, y_preds)
print(report)

Which outputs:

              precision    recall  f1-score   support

           0       0.87      0.70      0.78       410
           1       0.94      0.98      0.96      2069

    accuracy                           0.93      2479
   macro avg       0.91      0.84      0.87      2479
weighted avg       0.93      0.93      0.93      2479

And I can now make predictions like this:

print(clf.predict(['I hate you']))
print(clf.predict_proba(['I hate you']))

Which outputs:

[1]
[[0.0749381 0.9250619]]

My Question

I have another dataset, let's call this Dataset B, independent to Dataset A. Dataset B is also an annotated text dataset on hate speech with the same columns.

What I want to do is to test the model I trained using Dataset A on Dataset B. I.e., I want to produce a classification report on my model's performance on Dataset B. Can I do that? If yes, how?

Clarification: I do not want to retrain my model using Dataset B. I want to see how good its performance is on the new dataset.


Similar Questions

There is this question that looks similar based on the title but does not ask what I'm asking.

There's is also this question which asks how to make predictions. However, I know how to make predictions, what I want to do is to test my trained model on a totally new dataset.

$\endgroup$
5
  • $\begingroup$ You can just iterate over the new samples (second dataset) and do "predict". Then, you compare the assigned labels with the ground truths. You can then calculate any score you want. $\endgroup$ – t_e_o Feb 10 '20 at 15:34
  • $\begingroup$ @t_e_o I was actually implementing this right now but it feels a bit 'dirty'. I would guess that there exists a more professional way to produce such a report. $\endgroup$ – Aventinus Feb 10 '20 at 15:40
  • $\begingroup$ Why not just clf.predict(Dataset_B_drop_label)? How is Dataset B different from X_test? $\endgroup$ – Ben Reiniger Feb 10 '20 at 16:30
  • $\begingroup$ @BenReiniger Because I'm stupid. It is as simple as that! Can you please write this an answer so that I can accept it? $\endgroup$ – Aventinus Feb 10 '20 at 16:47
  • $\begingroup$ It seems then that the second question you linked actually does answer your question. I'll vote to close as duplicate, then. $\endgroup$ – Ben Reiniger Feb 11 '20 at 3:10