1
$\begingroup$

I am performing binary text classification. I have to classify a tweet 0 if neutral and 1 if hate speech.

So as general thumb rule i preprocessed my data. create term document frequency and After removing sparse terms i divide my data into train and test. I train my model using random forest and logistic regression and it worked fine.

set.seed(123)
tweetRand = randomForest(label ~ ., data = train_sparse, importance=TRUE, nTree=500 )
randPridct = predict(tweetRand, newdata = test_sparse)
table(test_sparse$label,randPridct >=0.5)

Its is working fine on test data which divided from raw content. But when i am running it on a new unseen data it is throwing an exception.

> predicrRand_test=predict(tweetRand, newdata=sparse_4testing)
Error in eval(predvars, data, env) : object 'run' not found

My understanding is that 'run' is a feature present in training but not in unseen test data and during my model training 'run' was included in tdm. In preprocessing of test , run was not in test tdm.

SO how should i deal with these situation. I am new to data science. Please help.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

It's perfectly fine to have different features in your training and testing partitions. If the two groups are randomly selected from the same population, one would expect, if there are many possible features, as is frequently the case in text classification, that differences would be observed. To handle this, the standard practice in the field is to train your model on the training partition, and then evaluate on the testing partition with any new features in that grouping being discarded. If you encounter the edge case of a test observation with no features after this procedure, it is common practice to use a heuristic to classify it, bypassing the model entirely (e.g., an observation with no features is classified as neutral).

Improve this answer
$\endgroup$
2
  • $\begingroup$ I have the same q as OP. If I understand correctly, you are suggesting to remove features from test data, that are not present in training. Trouble is, the fit() and predict() fns of sklearn models need the same dimension for train and test features. Further, with text processing, vectorization does not have the actual words anymore. So how exactly to get the intersection for both train and test features? $\endgroup$
    – Vaibhav
    May 4, 2018 at 3:05
  • $\begingroup$ Excellent question, @Vaibhav. If you're using something like binary feature modeling, you would have a 0 in the vector element associated with the missing feature in the test data that was observed in the training data. Generating the vector is done via a function you write...say, a python dictionary wherein the feature is the key and the vector element is the value. Then it's a simple step to transform into a deterministic vector. There may be more efficient ways of doing this in sklearn, but I like the old school approach sometimes. $\endgroup$
    – Kyle.
    May 31, 2018 at 16:58
-1
$\begingroup$

You are right it is probably due to new features in the unseen test data. This is data science 101, you should have same features in the test/train datasets.

My guess is that it is due to new levels of a categorical variable not present in the train data, but present in the unseen test data and this comes up as an error in the randomforest scoring. To take care of it you should select a randomized sample for test/train from your population and then harmonize the levels of the categorical variable using the techniques given here:

Harmonize levels of a categorical variable in test/train

Improve this answer
$\endgroup$
1
  • $\begingroup$ I disagree. In text classification it is quite common for there to be different features in the training and test partitions! $\endgroup$
    – Kyle.
    Apr 30, 2018 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.