I am performing binary text classification. I have to classify a tweet 0 if neutral and 1 if hate speech.
So as general thumb rule i preprocessed my data. create term document frequency and After removing sparse terms i divide my data into train and test. I train my model using random forest and logistic regression and it worked fine.
set.seed(123) tweetRand = randomForest(label ~ ., data = train_sparse, importance=TRUE, nTree=500 ) randPridct = predict(tweetRand, newdata = test_sparse) table(test_sparse$label,randPridct >=0.5)
Its is working fine on test data which divided from raw content. But when i am running it on a new unseen data it is throwing an exception.
> predicrRand_test=predict(tweetRand, newdata=sparse_4testing) Error in eval(predvars, data, env) : object 'run' not found
My understanding is that 'run' is a feature present in training but not in unseen test data and during my model training 'run' was included in tdm. In preprocessing of test , run was not in test tdm.
SO how should i deal with these situation. I am new to data science. Please help.