In the R code below, I included the sentences when looking to compare the manually classified with lexicon dictionary results by positive, negative and neutral (in matrixdata1), the algorithms results for the model produces different outcome in the tables, which is good. However, when executing..
results2 = classify_models(container2, models)
..when feeding in new data
(matrixdata2) against the model it produces an error message:
Error in predict.svm(model, container@classification_matrix, prob = TRUE, : test data does not match model !
In checking the datasets, I understand the train set's sentences used to create the model contains specific words, but the new data fed against the model include new words not recognised in the train set. I did create a few sentences that contained only words that appeared in the creation of the model. I fed in this few sentences with its labelled sentiment (new data) in against the model, but it still produced the same error message above. I do not understand why this is the case as these words are recognised in the trainset. However, when I used one of the same sentences in the new data to feed in against the model, it worked, so from what I can tell is if the sentence does not exactly match whats in the trainset, then it produces the error. I am still unsure how to adapt the R code to rectify the issue.
Please can you help me adapt the the R code below to overcome the error?
#Load Libraries library(RTextTools) #RTextTools available for 3.4.1 library(e1071) library(gmodels) setwd(directory/path) text= read.csv("matrixdata1.csv", header = FALSE) # build dtm matrix= create_matrix(text[,1:2]) mat = as.matrix(matrix) # build the data to specify response variable, training set, testing set. container = create_container(mat, as.numeric(as.factor(text[,3])), trainSize=1:672, testSize=673:840,virgin=FALSE) models = train_models(container, algorithms=c("MAXENT" , "SVM", "RF", "BAGGING", "TREE"), set_heldout = 168) #container1 results1 = classify_models(container, models) text2 = read.csv("matrixdata2.csv", header = FALSE) matrix2= create_matrix(text2[,1:2]) mat2 = as.matrix(matrix2) container2 = create_container(mat2, labels=NULL, trainSize=1:500,testSize=NULL, virgin=TRUE) #Results from feeding in new data against the model #When running this code below, it produces the error message outlined above in the description of the problem. results2 = classify_models(container2, models)