2
$\begingroup$

I am trying to classify set of text files whether it belong to category A or category B using KNN algorithm

Here is the error that i get

> knn.pred <- knn(tdm.stack.nl, tdm.stack.nl_test, tdm.cand)

Error in knn(tdm.stack.nl, tdm.stack.nl_test, tdm.cand) : dims of 'test' and 'train' differ

 > dim(tdm.stack.nl)  
    [1]  184 1599  
    > dim(tdm.stack.nl_test)  
    [1]   1 992

As obvious the number of words used by training dataset is different from train. How can I avoid this?

I want to mention that this error i get when i divide the training and test dataset manually to (70:30) ratio. However i dont get this error if i divide the training and test dataset using samlpling method.

> # create hold-out   
> set.seed(500)  
> train.idx <- sample(nrow(tdm.stack), ceiling(nrow(tdm.stack) * 0.7))   
> test.idx <- (1:nrow(tdm.stack))[-train.idx]   
> 
> # create model - knn clustering
> tdm.cand <- tdm.stack[, "targetcandidate"]
> tdm.stack.nl <- tdm.stack[, !colnames(tdm.stack) %in% "targetcandidate"]
> 
> # set up model
> knn.pred <- knn(tdm.stack.nl[train.idx,], tdm.stack.nl[test.idx,],    tdm.cand[train.idx])  
> dim(tdm.stack.nl[train.idx,])  
[1]  129 1599  
> dim(tdm.stack.nl[test.idx,])  
[1]   55 1599

However this method will not work for me as I want to classify a document in a real time using the model that was built before.

Please help. Kindly let me know if you need any other info. Arun

Improve this question
$\endgroup$
1
  • $\begingroup$ I am facing the same issue which the same data set as above library(tm) library(plyr) #for keeping the text as-is options(stringasFactors=F) #dicrectory location team<-c("AA","BB") storedin<-"XXXX" #pre-process; one corpus for each candidate cleanCorpus <- function(corpus){ mycorpus <- tm_map(corpus, removePunctuation) mycorpus <- tm_map(mycorpus,stripWhitespace) mycorpus <- tm_map(mycorpus, PlainTextDocument) mycorpus <- tm_map(mycorpus,removeWords, stopwords("english")) return(mycorpus) } #tdm mytdm<-function(teamember,storedpath){ emaillocation<-sprintf("%s/%s",storedpath,teamember) newcorp $\endgroup$
    – rohit kumar
    Commented Sep 12, 2016 at 14:51

2 Answers 2

Reset to default
1
$\begingroup$

You are avoiding the error with sampling purely by chance (the words are common). Simplest way would be to merge training and test, construct a tdm on a joined set, separate into training and test again and then purge some columns (the constant ones in the train set, as they correspond to words occurring in test only => useless for training).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer. I will try this. In the meanwhile, I have a doubt. Lets say that i want to build the above classification model now, and reuse that later to classify the documents later, how can i do that? Given that we have to perform the TDM as a single step for both training and test dataset, will saving a model and reusing it will work? $\endgroup$
    – Arun
    Commented Aug 31, 2015 at 18:18
  • $\begingroup$ @Arun: In Python, you can fit the transformation and store it as a pickle for later use. Closest counterpart in R I can think of is dumping the fitted model (TDM) in an .RData file and then filtering a new test set (matching the column names) prior to use. Crude, but it might get the job done. $\endgroup$
    – kpb
    Commented Sep 2, 2015 at 8:40
0
$\begingroup$

add 'Drop' in cl. It will work.

Example: cl = train_labels[,1, drop = TRUE] knn(train_points, test_points, cl, k = 5)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.