1
$\begingroup$

I have a dataset born to solve a classification problem. Due to the imbalances of the Y, i choose to move to an anomaly detection task. Should I use the Y i have inside the anomaly detection model as a features? Is it an overfitting Risk?

$\endgroup$
6
  • $\begingroup$ If you have imbalanced data you can use oversampling or under sampling techniques to deal with it. $\endgroup$
    – Ethan
    Commented Dec 1, 2020 at 2:46
  • $\begingroup$ It doesn't improve my results, for that reason I use anomaly detection $\endgroup$
    – Daniele
    Commented Dec 1, 2020 at 7:50
  • $\begingroup$ How large is your dataset? $\endgroup$
    – Ethan
    Commented Dec 1, 2020 at 7:51
  • $\begingroup$ 500k x 20. The original labels instead are the following: class a : 480k, class b: 15k, class c: 5k $\endgroup$
    – Daniele
    Commented Dec 1, 2020 at 8:41
  • $\begingroup$ To me it is surprising that you would find no improvement from resampling with a data set as large as 500k. $\endgroup$
    – Ethan
    Commented Dec 1, 2020 at 16:54

1 Answer 1

0
$\begingroup$

It simply depends on what is the goal of the task:

  • If the final goal is still to predict Y after detecting anomalies (i.e. probably using the output of anomaly detection as a feature), then Y cannot be used since it wouldn't be available in a realistic test set.
  • If it's just a completely different task in which Y is available as an input, then why not use it.

With 500k instances, A single additional variable with 3 possible values has an extremely low risk of causing overfitting.

Note that since classification didn't work, it's likely that there is little relationship between the features and Y (otherwise there was some mistake in the classification experiment).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.