6
$\begingroup$

Given some dataset for prediction,

for eg say I have different housing price prediction dataset:

dataset 1 : 100 training and 100 testing sample, 50 feature

dataset 2 : 100 training and 100 testing sample, 120 feature

dataset 3 : 1000 training and 1000 testing sample, 50 feature

dataset 4 : 1000 training and 1000 testing sample, 5000 feature

how should I choose the best methods for estimating the unknown parameters ( predict price) in a linear regression model from the following for each of these dataset?

  • Ordinary least squares

  • Stepwise regression

  • Principal component regression

  • Partial least squares regression

Should I experiment with each of these one by one and compare the results or is there any rule of thump on when to use each of them based on the dataset ?

Please help

Improve this question
$\endgroup$
5
  • $\begingroup$ It seems that least squares estimator and stepwise regression may fail if the variables are almost collinear. $\endgroup$
    – Sreejithc321
    Jun 1 '15 at 13:23
  • $\begingroup$ Sorry- this should be a comment. With prediction we don't need worry about collinearity. Ideally, you should cut out the predictors that measure the same thing. For prediction I would say OLS will do the trick if your response is continuous. As far as what predictors to include in the model, well, you can put them all if you want but you run the risk of overfitting. But generally with prediction, the more predictors the better the prediction. $\endgroup$
    – Lauren Goodwin
    Jun 1 '15 at 14:51
  • $\begingroup$ You did not include a fundamental piece of information: how many observations do you have? :) $\endgroup$
    – stochazesthai
    Jun 1 '15 at 16:37
  • 1
    $\begingroup$ What is meant by "unknown parameters"? Is your question about how to choose the final set of features using which you would carry out "ordinary least squares regression"? Consider editing your question description with this information as well as updating the title of the question. $\endgroup$
    – Nitesh
    Jun 1 '15 at 21:25
  • $\begingroup$ Hey If you get your answer please suggest me some steps , I am doing stock prediction with 25 unknown feature(Parameter) , weight and rate for 5 windo (day-2,day-1,intraday,day+1,day+2) ... I am doing with python .please suggest me something . $\endgroup$
    – user3762198
    Dec 1 '15 at 21:12
3
$\begingroup$

After the Data Munging, this is the most difficult task on a prediction model. However, in order to answer it, we need more details. What do you mean by the "best model"? Do you want accuracy and long training time? Do you need something really fast with lower accuracy? Something between the two of them? What are your features? Have you just taken them or have you created new features from them?

In any case, I suggest that you should spend some time and read this perfect tutorial from Microsoft about Machine Learning. A part of the tutorial to understand what I mean:

Regression

  1. Ordinal regression: Data in rank ordered categories
  2. Poisson regression: Predicting event counts
  3. Fast forest quantile regression: Predict a distribution
  4. Linear regression: Fast training, linear model
  5. Bayesian linear regression: Linear model, small data sets
  6. Neural Network regression: Accuracy, long training time
  7. Decision forest regression: Accuracy, fast training
  8. Boosted decision tree regression: Accuracy, fast training

When I have a similar question and I don't know which one to choose, I usually end up to 3-4 different algorithms based on the cheat sheet of Microsoft or the one from the scikit-learn, try them all and choose one or two of them with the best results.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.