1
$\begingroup$

I'm using a set of features, says $X_1, X_2, ..., X_m $, to predict a target value $Y$, which is a continuous value from zero to one.

At first, I try to use a linear regression model to do the prediction, but it does not perform well. The root-mean-squared error is about 0.35, which is quite high for prediction of a value from 0 to 1.

Then, I have tried different models, e.g., decision-tree-based regression, random-forest-based regression, gradient boosting tree regression and etc. However, all of these models also do not perform well. (RMSE $\approx $0.35, there is not significant difference with linear regression)

I understand there are many possible reasons for this problem, such as: feature selection or choice of model, but maybe more fundamentally, the quality of data set is not good.

My question is: how can I examine whether it is caused by bad data quality?

BTW, for the size of data set, there are more than 10K data points, each of which associated with 105 features.

I have also tried to investigate importance of each feature by using decision-tree-based regression, it turns out that, only one feature (which should not be the most outstanding feature in my knowledge to this problem) have an importance of 0.2, while the rest of them only have an importance less than 0.1.

|improve this question
$\endgroup$
3
$\begingroup$

First, it sounds like your choice of model selection is a problem here. Your outputs are binary-valued, not continuous. Specifically you may have a classification problem on your hands rather than a traditional regression problem. My first recommendation would be to try a simple classification approach such as logistic regression or linear discriminant analysis.

Regarding your suspicions of bad data, what would bad data look like in this situation? Do you have reason to suspect that your $X$ values are noisy or that your $y$ values are mislabeled? It is also possible that there is not a strong relationship between any of your features and your targets. Since your targets are binary, you should look at histograms of each of your features to get a rough sense of the class conditional distributions, i.e. $p(X_1|y=1)$ vs $p(X_1|y=0)$. In general though, you will need to be more specific about what "bad data" means to you.

|improve this answer
$\endgroup$
  • 1
    $\begingroup$ I have change the description of ambiguous description of Y, actually it is a continuous value from zero to one, not a binary value of zero or one. $\endgroup$ – ice_lin Feb 4 '15 at 7:32
  • $\begingroup$ In that case I would suggest to examine scatter plots between each of your X variables versus y (instead of histograms). Additionally since your target data is bound between 0 and 1, you may still want to consider trying logistic regression. $\endgroup$ – Ryan J. Smith Feb 4 '15 at 7:41
  • 2
    $\begingroup$ @RyanJ.Smith you can't do logistic regression on real values. You can do beta regression on numbers bound to the [0,1] interval, or you can do linear regression on the logit transform $\endgroup$ – Ben Allison Feb 4 '15 at 13:07
  • $\begingroup$ Good catch, Ben. I had forgotten that though the output from logistic regression is continuous, the response is originally assumed to be Bernoulli (binary). $\endgroup$ – Ryan J. Smith Feb 4 '15 at 17:53
  • $\begingroup$ +1 for "It is also possible that there is not a strong relationship between any of your features and your targets." This is the first explanation that would come to my mind. Prediction is often hard. $\endgroup$ – S. Kolassa - Reinstate Monica Feb 4 '15 at 20:02
1
$\begingroup$

How many features do you have?

Is quite unlikely that ALL the features are bad. So you could regress with a different number of features.

For example, do one pass with all the features, then take one out (usually X_m) so you have m-1 features. Keep doing this so you can take out uninformative features.

Also, I would recommend you calculating P-Values to see whether your regessors are significative are informative.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.