Get a prediction from the new data inputted against the model, but an error is produced, how to adapt the R code for it to work?

Question

In the R code below, I included the sentences when looking to compare the manually classified with lexicon dictionary results by positive, negative and neutral (in matrixdata1), the algorithms results for the model produces different outcome in the tables, which is good. However, when executing..

results2 = classify_models(container2, models)

..when feeding in new data (matrixdata2) against the model it produces an error message:

Error in predict.svm(model, container@classification_matrix, prob = TRUE,  : 
  test data does not match model !

In checking the datasets, I understand the train set's sentences used to create the model contains specific words, but the new data fed against the model include new words not recognised in the train set. I did create a few sentences that contained only words that appeared in the creation of the model. I fed in this few sentences with its labelled sentiment (new data) in against the model, but it still produced the same error message above. I do not understand why this is the case as these words are recognised in the trainset. However, when I used one of the same sentences in the new data to feed in against the model, it worked, so from what I can tell is if the sentence does not exactly match whats in the trainset, then it produces the error. I am still unsure how to adapt the R code to rectify the issue.

Please can you help me adapt the the R code below to overcome the error?

#Load Libraries
library(RTextTools) #RTextTools available for 3.4.1
library(e1071)
library(gmodels)

setwd(directory/path)
text= read.csv("matrixdata1.csv", header = FALSE)

# build dtm
matrix= create_matrix(text[,1:2])
mat = as.matrix(matrix)

# build the data to specify response variable, training set, testing set.
container = create_container(mat, as.numeric(as.factor(text[,3])),
                             trainSize=1:672, testSize=673:840,virgin=FALSE)

models = train_models(container, algorithms=c("MAXENT" , "SVM", "RF", "BAGGING", "TREE"), set_heldout = 168) 

#container1
results1 = classify_models(container, models)

text2 = read.csv("matrixdata2.csv", header = FALSE) 
matrix2= create_matrix(text2[,1:2])
mat2 = as.matrix(matrix2)

container2 = create_container(mat2, labels=NULL, trainSize=1:500,testSize=NULL, virgin=TRUE)

#Results from feeding in new data against the model
#When running this code below, it produces the error message outlined above in the description of the problem.
results2 = classify_models(container2, models)

matrixdata1.csv on GitHub

matrixdata2.csv on GitHub

Answer 1

This is a quite common mistake, you transformed the test and training data separately which messes with factor levels.

There are multiple solutions to this:

1) Create a common transformer

If you there is no "new" data but simply a test set (e.g. like in kaggle competitions or similar problems) you should create a function that transforms / tidies ALL data at once and only split the data after it already has the proper form for the modelling (including OHE, etc.).

2) Save your factor levels!

For all other case you should save your factor levels and create a transformer accordingly. Here is a code example:

text_levels <- levels(train$text)

levels(test$text) <- text_levels

Realize also, that any word or factor level not present in the data you train your model on cannot be considered in the model and therefore is useless! All above solutions simply add the appropriate factor levels from the training model if they are missing in the test data.

If you have new words in the test data you will simply need to drop them before transforming the data as they are not used in the prediction model.