0
$\begingroup$

In the R code below, I included the sentences when looking to compare the manually classified with lexicon dictionary results by positive, negative and neutral (in matrixdata1), the algorithms results for the model produces different outcome in the tables, which is good. However, when executing..

results2 = classify_models(container2, models)

..when feeding in new data (matrixdata2) against the model it produces an error message:

Error in predict.svm(model, container@classification_matrix, prob = TRUE,  : 
  test data does not match model !

In checking the datasets, I understand the train set's sentences used to create the model contains specific words, but the new data fed against the model include new words not recognised in the train set. I did create a few sentences that contained only words that appeared in the creation of the model. I fed in this few sentences with its labelled sentiment (new data) in against the model, but it still produced the same error message above. I do not understand why this is the case as these words are recognised in the trainset. However, when I used one of the same sentences in the new data to feed in against the model, it worked, so from what I can tell is if the sentence does not exactly match whats in the trainset, then it produces the error. I am still unsure how to adapt the R code to rectify the issue.

Please can you help me adapt the the R code below to overcome the error?

#Load Libraries
library(RTextTools) #RTextTools available for 3.4.1
library(e1071)
library(gmodels)

setwd(directory/path)
text= read.csv("matrixdata1.csv", header = FALSE)

# build dtm
matrix= create_matrix(text[,1:2])
mat = as.matrix(matrix)

# build the data to specify response variable, training set, testing set.
container = create_container(mat, as.numeric(as.factor(text[,3])),
                             trainSize=1:672, testSize=673:840,virgin=FALSE)

models = train_models(container, algorithms=c("MAXENT" , "SVM", "RF", "BAGGING", "TREE"), set_heldout = 168) 

#container1
results1 = classify_models(container, models)

text2 = read.csv("matrixdata2.csv", header = FALSE) 
matrix2= create_matrix(text2[,1:2])
mat2 = as.matrix(matrix2)

container2 = create_container(mat2, labels=NULL, trainSize=1:500,testSize=NULL, virgin=TRUE)

#Results from feeding in new data against the model
#When running this code below, it produces the error message outlined above in the description of the problem.
results2 = classify_models(container2, models)

matrixdata1.csv on GitHub

matrixdata2.csv on GitHub

$\endgroup$
1
$\begingroup$

This is a quite common mistake, you transformed the test and training data separately which messes with factor levels.

There are multiple solutions to this:

1) Create a common transformer

If you there is no "new" data but simply a test set (e.g. like in kaggle competitions or similar problems) you should create a function that transforms / tidies ALL data at once and only split the data after it already has the proper form for the modelling (including OHE, etc.).

2) Save your factor levels!

For all other case you should save your factor levels and create a transformer accordingly. Here is a code example:

text_levels <- levels(train$text)

levels(test$text) <- text_levels

Realize also, that any word or factor level not present in the data you train your model on cannot be considered in the model and therefore is useless! All above solutions simply add the appropriate factor levels from the training model if they are missing in the test data.

If you have new words in the test data you will simply need to drop them before transforming the data as they are not used in the prediction model.

$\endgroup$
5
  • $\begingroup$ Thank you for the information, much appreciated. Even if I am introducing unseen (new data) data it has to contain the same number of columns, is that what you are outlining above? Also, with the new words in the test data, rather than simply removing rows before transformation, is there a way the ML process can take these into consideration? $\endgroup$ – jr134 Apr 22 '20 at 10:20
  • $\begingroup$ @jr134 Not the same number of columns, the SAME columns! Most often the difference stems from one-hot-encoding (OHE) and different factor levels. So c(a,b) in train will lead to two columns a/b in the model which fails if test has only c(a) because it will not create column b. Therefore you have to add factor levels so that OHE still produces all columns from the train set even if they are empty! $\endgroup$ – Fnguyen Apr 22 '20 at 10:25
  • $\begingroup$ @jr134 Unseen data can only be considered in the ML process based on assumptions. So if a do a word prediction, I could group low frequency/strange words into a factor level "other" and use that in my model. I could then make the assumption that all new words in unseen data belong into that same column and will have the same effect on my prediction. For most cases this seems risky and will lead to false positives. If you have the founded believe that unseen data is very different from your train data you simply don't have enough data yet to train a good model! $\endgroup$ – Fnguyen Apr 22 '20 at 10:27
  • $\begingroup$ I am new to this, so your information is appreciated to help further my understanding. Is it possible to overcome the issue with either of these options below: - Point the ML to consult a dictionary of words to look up and return a result - For the machine learning process to identify ones that match and ones that do not match simply say I don't know response from ML. $\endgroup$ – jr134 Apr 22 '20 at 11:32
  • $\begingroup$ @jr134 I'd need a bit more information about your model and intend. As I'm not quite sure what you mean with the dictionary and what each of your input factors are. However I can point you to a pretty well annotated text prediction notebook I published on Kaggle solving a similar problem:kaggle.com/fnguyen/… $\endgroup$ – Fnguyen Apr 22 '20 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.