1
$\begingroup$

I have a data set consisting of a bunch of predictors (mostly unbounded or positive real numbers) and a single response variable that I wish to predict. The response is typically exactly zero -- around 90% of the time. I have tried modelling this using standard Gaussian process methods as well as random forests. However, in both cases (although moreso when using random forests) the model seems to handle the data poorly, usually predicting a non-zero response. Now, if the predicted responses were in fact very close to zero I could just set a cut-off below which the values would be rounded to zero, but they are significantly non-zero in many cases.

My idea for a solution is to train two models: a classification model trained on the entire training-set that predicts whether a variable is zero or non-zero, and a regression model trained only on the rows in the training set with a non-zero response. I would then first use the classification model to predict which observations have a response that is exactly zero, and subsequently use the regression model to predict the value of the non-zero responses.

Is this a sound way to solve the described problem? Does this sort of model have a name? Are there better ways to do this?

$\endgroup$
  • $\begingroup$ Pipeline sounds good and maybe general adversarial networks GAN as well. $\endgroup$ – Georg Heiler Apr 19 '17 at 6:19
3
$\begingroup$

This sounds entirely reasonable, and the usual name for this structure I have heard for this is just "pipeline" which also applies to other system-feeds-next-system structures - it might also be "machine learning pipeline" or "data processing pipeline".

There are ways to assess performance of a ML pipeline:

  • You can of course compare the final accuracy or loss value, with the simpler model. Has turning the model into a more complex multi-stage one actually improved things? Sadly nothing is guaranteed, although I would be hopeful in your case initially - in part because you could apply adjustments available to classifier models used to deal with class imbalance issues.

  • You can decide which part of the pipeline will gain you the most benefit by switching between pipeline-so-far input to each unit and perfect input from the training data. Then you can see how much incremental difference is possible by perfecting that unit in the pipeline.

In your case you have a two stage pipeline, so you can check whether it is worth focusing more effort on the classifier or regression parts by comparing the incremental improvements between:

  1. The unadjusted output of the whole pipeline run end-to-end.

  2. The output of the regression (or zero) assuming that the classifier was perfect.

  3. A perfect score.

Whichever of the two differences gives you the largest difference (2) - (1), or (3) - (2) points at work being most rewarded for working on the classifier or regression stage respectively.

You can see a worked example of this per-stage analysis in Advice for Applying Machine Learning (slides 21, 22), amongst other places.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.