-1
$\begingroup$

I have a dataset with some features. Over one feature, there are some values that appear only once. Can I safely remove these rows, or do they need to be kept ?

$\endgroup$
1
  • 2
    $\begingroup$ Are you expecting those values to appear again during prediction? You might want to experiment with dropping the column instead, if it is not strongly predictive of your target (assuming supervised learning). For a detailed answer, it may be worth giving more context to your problem, and more specifics of the data, because there could be a way to handle the feature. $\endgroup$ Jul 2, 2017 at 13:08

1 Answer 1

1
$\begingroup$

If you have a categorical feature of which one level appears only once in the data, then an algorithm that estimates its parameters with that data will assign all the residual for that observation to that feature (unless you have some regularization in place).

Thus, that would probably be a very incorrect estimate, as it will just try to correct all the errors from the other parameters, and as such will probably not generalize to newer observations. This of course depends on the data that you have, the number of features, the goodness of fit (e.g. if your goodness of fit or classification accuracy is something like 99% then perhaps it might not be so bad to include that feature/row, especially if you are using high regularization parameters) and other factors, but in most cases you’d be better off leaving that feature out, or if possible, merging that into another category.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.