1
$\begingroup$

My training data comes in batches. Sometimes, new batches (completely new samples) come with new columns that are not in old batches, or they may be missing some of the old columns.

For example, suppose there are two ingestions. In the 1st ingestion, we have ETL on a set of fields. In the 2nd ingestion, we have added a new field and we are not allowed to ingest and update the old records again (they may have been deleted for good).

Ideally, I want to train a classifier using all batches of data. What kind of algorithms would perform well under this scenario.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

A tree-based algorithm can do that.

The point is that you need to train the model with the union of the possible columns that can exist the different batches.

Moreover you need to account for missing values so that the model can learn to recognize a missing and handle them: you need to recode the missings with a proper value, for example you can create a new level for categorical variables and recode the numerical in the standard way (zero, mean, extreme value, etc.)

Improve this answer
$\endgroup$
3
  • $\begingroup$ Suppose we can replace missing values, do trees have an advantage over other algos? $\endgroup$
    – kakarukeys
    Commented Mar 26, 2019 at 10:31
  • $\begingroup$ Suppose we can't, is there an algo that can take records of varying dimension? $\endgroup$
    – kakarukeys
    Commented Mar 26, 2019 at 10:31
  • $\begingroup$ You better apply some preprocessing after the ETL to create a dataset that suits the model rules. on how to handle missings take a look here stats.stackexchange.com/questions/103500/… $\endgroup$
    – VD93
    Commented Mar 26, 2019 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.