Kmeans cluster validation when I have labeled test data

Question

I'm trying to implement the unsupervised k-means algorithm for sentiment analysis of imdb movie dataset created by stanford.
The steps that I followed is :
1) Load the comments
2) Apply tokenization and stemmetion ,use tf-idf algo to create tfidf matrix.
3) Use k-means algo to divide the data into 2 clusters.

My problem is how do I validate the the clusters
I have labeled test data. I want to check if all the negative examples go in one cluster and all the positive examples go in another cluster.

For this I used the scikits predict() command. But the parameters that need to be provided to the predict() command should be same as the parameters of fit() command. But as tfidf matrix that is passed has the dimensions dependent on the data passed.

Answer 1

In order to keep the same dimension for both training & testing set, you need apply your fit on the training set with you tokenization, tf-idf etc.. You need to apply same treatments as you do on training, on the testing set and then this will be okay :

from sklearn.feature_extraction.text import CountVectorizer

corpus_train = [
    'This is the first document.',
    'This document is the second document.'
]
corpus_test = [
    'And this is the third one.',
    'Is this the first document?',
]

vectorizer = CountVectorizer()
X_train = vectorizer.fit_transform(corpus_train)
print(vectorizer.get_feature_names())

# Returns
['document', 'first', 'is', 'second', 'the', 'this']

So your training set looks like this :

print(X_train.toarray())
# Returns
[[1 1 1 0 1 1]
 [2 0 1 1 1 1]]

print(X_train.shape)
# Returns
(2, 6)

Then just transform your testing set with the same vectorizer :

X_test = vectorizer.transform(corpus_test)
print(X_test.toarray())
# Returns 
[[0 0 1 0 1 1]
[1 1 1 0 1 1]]

print(X_test.shape)
# Returns
(2, 6)

So you get your dimension of 6 (the number of features) for both training & testing set :)