How to use two different datasets as train and test sets?

Question

Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. The problem that I've encountered, now that I'm trying to make my own classification algorithm (the text sends a positive/negative message) regards the training and the testing datasets. In all the examples that I've found, only one dataset is used, a dataset that is later split into training/testing. I have two datasets, and my approach involved putting together, in the same corpus, all the texts in the two datasets (after preprocessing) and after, splitting the corpus into a test set and a training set.

datasetTrain = pd.read_csv('train.tsv', delimiter = '\t', quoting = 3)
datasetTrain['PN'].value_counts()

datasetTest = pd.read_csv('test.tsv', delimiter = '\t', quoting = 3)
datasetTest['PN'].value_counts()

corpus = []
y = []

# some preprocessing
    y.append(posNeg)
    corpus.append(text)

from sklearn.feature_extraction.text import TfidfVectorizer
transf = TfidfVectorizer(stop_words = stopwords, ngram_range = (1,1), min_df = 5, max_df = 0.65)
X = transf.fit_transform(corpus).toarray()

# Splitting the dataset into the Training set and Test set
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.11, random_state = 0)

The reason why I've done this is because I'm working with the Bag of Words model and if I'm creating from the beginning X_train and X_test (y_train, y_test respectively) and not using the splitting function, I get an error when running the classification algorithm:

X_train = transf.fit_transform(corpustrain).toarray()
X_test = transf.fit_transform(corpustest).toarray()

...

classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)

ValueError: Number of features of the model must match the input. Model n_features is 2770 and input n_features is 585

I'm kind of new at this and I was wondering if anyone could please guide me in the right direction?

Answer 1

You may want to use a pipeline to do this operation. Specifically, you do NOT want to train the TFIDFVectorizer the entire corpus- doing so gives your model hints about what features may be in the test set that don't exist in the training set- a concept frequently referred to as "leakage" or "data snooping".

The correct pattern is:

transf = transf.fit(X_train)
X_train = transf.transform(X_train)
X_test = transf.transform(X_test)

Using a pipeline, you would fuse the TFIDFVectorizer with your model into a single object that does the transformation and prediction in a single step. It's easier to maintain a solid methodology within that pattern.

In the example code, you're both fitting and transforming in the same step fit_transform, which is creating different features each time and is the source of your error.