Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. The problem that I've encountered, now that I'm trying to make my own classification algorithm (the text sends a positive/negative message) regards the training and the testing datasets. In all the examples that I've found, only one dataset is used, a dataset that is later split into training/testing. I have two datasets, and my approach involved putting together, in the same corpus, all the texts in the two datasets (after preprocessing) and after, splitting the corpus into a test set and a training set.
datasetTrain = pd.read_csv('train.tsv', delimiter = '\t', quoting = 3) datasetTrain['PN'].value_counts() datasetTest = pd.read_csv('test.tsv', delimiter = '\t', quoting = 3) datasetTest['PN'].value_counts() corpus =  y =  # some preprocessing y.append(posNeg) corpus.append(text) from sklearn.feature_extraction.text import TfidfVectorizer transf = TfidfVectorizer(stop_words = stopwords, ngram_range = (1,1), min_df = 5, max_df = 0.65) X = transf.fit_transform(corpus).toarray() # Splitting the dataset into the Training set and Test set from sklearn.cross_validation import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.11, random_state = 0)
The reason why I've done this is because I'm working with the Bag of Words model and if I'm creating from the beginning X_train and X_test (y_train, y_test respectively) and not using the splitting function, I get an error when running the classification algorithm:
X_train = transf.fit_transform(corpustrain).toarray() X_test = transf.fit_transform(corpustest).toarray() ... classifier.fit(X_train, y_train) y_pred = classifier.predict(X_test) ValueError: Number of features of the model must match the input. Model n_features is 2770 and input n_features is 585
I'm kind of new at this and I was wondering if anyone could please guide me in the right direction?