1
$\begingroup$

I have a dataset with label noise which I wan't to clean with majority/consensus vote filtering. This will mean I will divide the data in K-Folds and train an ensemble model. Than using the predictions on the data I will remove rows, which are missclassified by most (majority voting) or all (consensus voting).

I have a few questions on which I can't find the answers elsewhere:

  • how to decide what models to use in the ensemble

  • the dataset is very imbalanced. Do I need to do upsampling in the majority voting?

  • do I do hyperparameter tuning in the different models, or just use standard settings?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

I have a few questions on which I can't find the answers elsewhere

It's probably because there is no simple answer to these three questions :)

I doubt there's any state of the art approach, in such cases I simply try to determine the answer to these questions empirically. Basically I create a list of hyper-parameters including the type of algorithm, the algorithm-specific hyper-parameters and any other potentially relevant option. The goal is to determine the optimal combination of values for the set of parameters. If practical I run all the combinations and select the best one. If not practical, I use a simple genetic algorithm to find an optimal combination. Of course it's suitable only if you have a dataset large enough and if the training/testing process is not too computer-intensive. You also need to be very careful about overfitting by using cross-validation and re-sampling.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Dear Erwan thanks a lot for your reply. The problem us I try to clean label noise with the majority voting. How would I evaluate the perfroamnce of the different set ups? $\endgroup$
    – Jeuszt
    Commented Jul 7, 2019 at 17:41
  • $\begingroup$ Oh right, I forgot that it's label noise so you can't evaluate against your labels. I think the only "clean" way is to annotate manually a small validation set as noise/not noise, i.e. make it a proper binary classification task that you can evaluate. if there is a massive imbalance between noise/not noise, you could bootstrap some minority class instances using a simple stage of majority voting. Any other option I can think of would amount to evaluating against the thing that you're trying to predict so it wouldn't be great. $\endgroup$
    – Erwan
    Commented Jul 7, 2019 at 19:46
  • $\begingroup$ Thanks for your reply :) $\endgroup$
    – Jeuszt
    Commented Jul 8, 2019 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.