R - Error in KNN - Test and training differ

I am trying to classify set of text files whether it belong to category A or category B using KNN algorithm

Here is the error that i get

> knn.pred <- knn(tdm.stack.nl, tdm.stack.nl_test, tdm.cand)


Error in knn(tdm.stack.nl, tdm.stack.nl_test, tdm.cand) : dims of 'test' and 'train' differ

 > dim(tdm.stack.nl)
[1]  184 1599
> dim(tdm.stack.nl_test)
[1]   1 992


As obvious the number of words used by training dataset is different from train. How can I avoid this?

I want to mention that this error i get when i divide the training and test dataset manually to (70:30) ratio. However i dont get this error if i divide the training and test dataset using samlpling method.

> # create hold-out
> set.seed(500)
> train.idx <- sample(nrow(tdm.stack), ceiling(nrow(tdm.stack) * 0.7))
> test.idx <- (1:nrow(tdm.stack))[-train.idx]
>
> # create model - knn clustering
> tdm.cand <- tdm.stack[, "targetcandidate"]
> tdm.stack.nl <- tdm.stack[, !colnames(tdm.stack) %in% "targetcandidate"]
>
> # set up model
> knn.pred <- knn(tdm.stack.nl[train.idx,], tdm.stack.nl[test.idx,],    tdm.cand[train.idx])
> dim(tdm.stack.nl[train.idx,])
[1]  129 1599
> dim(tdm.stack.nl[test.idx,])
[1]   55 1599


However this method will not work for me as I want to classify a document in a real time using the model that was built before.

• I am facing the same issue which the same data set as above library(tm) library(plyr) #for keeping the text as-is options(stringasFactors=F) #dicrectory location team<-c("AA","BB") storedin<-"XXXX" #pre-process; one corpus for each candidate cleanCorpus <- function(corpus){ mycorpus <- tm_map(corpus, removePunctuation) mycorpus <- tm_map(mycorpus,stripWhitespace) mycorpus <- tm_map(mycorpus, PlainTextDocument) mycorpus <- tm_map(mycorpus,removeWords, stopwords("english")) return(mycorpus) } #tdm mytdm<-function(teamember,storedpath){ emaillocation<-sprintf("%s/%s",storedpath,teamember) newcorp – rohit kumar Sep 12 '16 at 14:51

You are avoiding the error with sampling purely by chance (the words are common). Simplest way would be to merge training and test, construct a tdm on a joined set, separate into training and test again and then purge some columns (the constant ones in the train set, as they correspond to words occurring in test only => useless for training).

• Thanks for the answer. I will try this. In the meanwhile, I have a doubt. Lets say that i want to build the above classification model now, and reuse that later to classify the documents later, how can i do that? Given that we have to perform the TDM as a single step for both training and test dataset, will saving a model and reusing it will work? – Arun Aug 31 '15 at 18:18
• @Arun: In Python, you can fit the transformation and store it as a pickle for later use. Closest counterpart in R I can think of is dumping the fitted model (TDM) in an .RData file and then filtering a new test set (matching the column names) prior to use. Crude, but it might get the job done. – kpb Sep 2 '15 at 8:40

add 'Drop' in cl. It will work.

Example: cl = train_labels[,1, drop = TRUE] knn(train_points, test_points, cl, k = 5)