0
$\begingroup$

My R scripts and my 5 source datasets can be found in my GitHub Repository for this project, and I originally found this source data on Kaggle. This set of source data includes 5 datasets with over 200 annual aggregate financial indicators for an average of 4k US stocks from 2014 to 2018.

I am doing a sort of general classification and regression modeling project to practice using many of the ML/SL models I have learned in my Master's program in Data Analytics Engineering and one of the reasons I chose this source dataset was that conceptually speaking, I thought having 5 different datasets on annual aggregate stock indicators would make it easier to train, validate, and test all of my models multiple times. But, practically speaking, I cannot figure out how to do this in R.

I have trained a Logit, a Partial Least Squares Discriminant Analysis, an Elastic Net via glmnet, a Neural Network, an Average of several Neural Networks model, a Support Vector Machine Classification model, and a K-Nearest Neighbors model all on the 2014 stock market data and used them to predict the performance of those stocks in 2015. But all of the classification performance metrics look terrible for most of them even though I employed cross validation in the estimation of several of them, so what I would like to do is take my trained estimates from 2014, and further train them on the 2015 data and use the re-trained estimates to predict their behavior in 2016 to see if my test set accuracy improves. How should I go about this practically speaking in R?

Just to illustrate one example, for my Logistic Regression model, I used the following code in R:

set.seed(100)          # use the same seed for every model
ftLogitC1 <- train(x = data2014c, y = class2014c, method = "glm",
                   metric = "ROC", preProcess = c("center", "scale"),
                   trControl = ctrl_C)
> ftLogitC1
Generalized Linear Model 
513 samples
110 predictors
  2 classes: 'Decrease', 'Increase' 
Pre-processing: centered (110), scaled (110) 
Resampling: Repeated Train/Test Splits Estimated (25 reps, 75%) 
Summary of sample sizes: 386, 386, 386, 386, 386, 386, ... 
Resampling results:
  ROC        Sens       Spec     
  0.5706867  0.6142857  0.5017544

# compare the expected classifications in 2015 to the observed classifications in 2015
LogitC1_predictions_for_2015 <- predict(ftLogitC1, data2015c)

# create a confusion matrix to show how well our Logit model's predictions fared
# by inspecting all of its classification model performance metrics
LogitC1_CFM <- confusionMatrix(data = LogitC1_predictions_for_2015, 
                               reference = class2015c, positive = "Increase")

And when I print out the above Confusion Matrix, I get an accuracy of only 0.395, a Kappa of -0.084, a True Positive Rate of 0.38, a True Negative Rate of 0.47, a Positive Predictive Value of 0.75, and an AUC of 0.613, which collectively is absolutely abysmal performance!

One important thing I should add here is that I have already asked an extremely similar question to this on Stack Overflow yesterday but have gotten no responses yet and I was not sure whether this sort of question would be more appropriate to ask here or there to be honest.

$\endgroup$
1
  • 1
    $\begingroup$ What is your question exactly ? It’s not clear you have a problem. Metrics in finance are known to be bad (imagine if everyone could predict the stock market after a ml course). If anything yours appear to be a bit high. $\endgroup$ Jan 8 at 23:35

1 Answer 1

1
$\begingroup$

Your code appears correct for this project. Additionally, the performance of the model appears acceptable for the nature of the project. It is very difficult to predict stock returns based on historical data. "Past performance is no guarantee of future results" is a standard financial prediction disclaimer.

$\endgroup$
4
  • $\begingroup$ I completely agree with you about predicting future stock market behavior. I am simply using this set of datasets and this script as an example because I was the member of our group which suggested this source data for the specific reason that I thought it would be easier to do cross validation if there are already 5 roughly equivalent datasets for you so that you don't have to use sample splitting multiple different times. Again, this is very straightforward conceptually speaking, I am just a very weak coder at the moment. $\endgroup$
    – Marlen
    Jan 9 at 19:06
  • $\begingroup$ allow me to ask a more specific question to see if this helps clarify. If I were to just combine data (only on companies in both years of course) for two years at the start to try to get slightly better predictions, that wouldn't be model tuning, it would just be getting more training data which is the classical statistics approach, not the data science & machine learning approach. I want to know how to use the next year's data afterwards to tune and re-estimate the predictions. That is literally what model tuning is according to my textbooks, except with new data. $\endgroup$
    – Marlen
    Jan 9 at 19:11
  • 1
    $\begingroup$ The main thing you want is to avoid time leakage (evaluating on data that happened before or during the train). One way would be to train on a given year and evaluate on the next for a start. For more advanced time series cv scheme see kaggle (exemple: kaggle.com/competitions/tabular-playground-series-sep-2022/…) $\endgroup$ Jan 9 at 19:29
  • $\begingroup$ @Icrmorin exactly, you understand what I am asking perfectly! So, what I had planned to do was start off by training on the 2014 data and using it to predict the 2015 data, then tune the predictions using the 2015 data and use the new models to predict the 2016 data. Seemed very straightforward and obviously consistent with my understanding of statistical learning, but I just could not figure out how to code this seemingly simple idea. $\endgroup$
    – Marlen
    Jan 10 at 8:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.