1
$\begingroup$

I am trying to forecast sector's 1 month forward return using macroeconomic variables (170 variables). I tried a few things:

For variable Selection

  1. Run regression with each one of the independent variables and dependent variable separately. Then, using independent variables whose p-values are less than 0.05 to:
  2. Run stepwise regression for variable selection
  3. Finally, run elastic net with best variables (chosen using the above steps i.e., step 1 and step 2)

My observation:
Results (r2) for test set largely depends on how data is split (I don't know why) and even in the best split, r2 is very less (as per my client) in range of 0.12-0.20. How can I either improve this result or conclude that it can't be improved with linear regression? I have attached a couple of diagnostic plots for independent variables which have the lowest p-values enter image description here enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

In general, it is very hard to prove a negative. So, concluding that a result cannot be somehow improved with some unspecified technique tends to be almost impossible. Often the best one can do is to follow state-of-the-art good practices and try out several sensible approaches. Then it is at least unlikely that the results can be easily improved.

For example, using p-value cut-offs from univariate regressions to eliminate variables and using stepwise model selection is known to be a poor practice, so I would assume that one could improve over your current results. For example, without knowing more details of your data, I would expect that elastic net without pre-screening variables with the hyperparameters selected using cross-validation (within the training set) would perform better on a hold-out test set in many cases.

Additionally, if you primarily care about predictive performance then linear regression is unlikely to achieve the best predictive performance. Some form of boosted tree or similar approach will likely be substantially better, variables may tend to be non-linearly related to the outcome. With some subject matter knowledge/good feature engineering you may also achieve pretty good results with penalized (i.e. ridge/elastic net or similar) linear regression - but perhaps the outcome needs to be transformed (e.g. to the log-scale), perhaps some predictors need transforming (e.g. also to the log-scale) and perhaps we can expect certain interactions to be potentially important.

It is also not clear how you are evaluating performance (cross-validation? a hold-out set?). Evaluating model performance on data that directly entered the modeling is likely to be overoptimistic, particularly when you have many potential predictors and presumably not a lot of data. You mentioned a split, which I speculated might be a train-test split, but then it seemed like you kept changing the split to find good performance metrics. If you do that, you can no longer trust the resulting performance metrics.

$\endgroup$
2
  • $\begingroup$ I tried (log, inv,exp) transformations on each indep variable, and then chose best variables by step wise regression. After choosing best transformed variables, I split data in test/ train for evaluating performance on hold-out set. Then I did a gridSearch on train set for finding best combination of (alpha, l1_ratio, max_iter) of elastic net. And finally tested the model on test set (r2 i am talking about). I understand doing different splits, i won't find good performance metrics. How should I interpret very varying results with different train/test splits? $\endgroup$
    – dgomzi
    Commented Jul 11, 2018 at 12:49
  • $\begingroup$ I suspect that shows that you have not a lot of data. $\endgroup$
    – Björn
    Commented Jul 11, 2018 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.