My R scripts and my 5 source datasets can be found in my GitHub Repository for this project, and I originally found this source data on Kaggle. This set of source data includes 5 datasets with over 200 annual aggregate financial indicators for an average of 4k US stocks from 2014 to 2018.
I am doing a sort of general classification and regression modeling project to practice using many of the ML/SL models I have learned in my Master's program in Data Analytics Engineering and one of the reasons I chose this source dataset was that conceptually speaking, I thought having 5 different datasets on annual aggregate stock indicators would make it easier to train, validate, and test all of my models multiple times. But, practically speaking, I cannot figure out how to do this in R.
I have trained a Logit, a Partial Least Squares Discriminant Analysis, an Elastic Net via glmnet, a Neural Network, an Average of several Neural Networks model, a Support Vector Machine Classification model, and a K-Nearest Neighbors model all on the 2014 stock market data and used them to predict the performance of those stocks in 2015. But all of the classification performance metrics look terrible for most of them even though I employed cross validation in the estimation of several of them, so what I would like to do is take my trained estimates from 2014, and further train them on the 2015 data and use the re-trained estimates to predict their behavior in 2016 to see if my test set accuracy improves. How should I go about this practically speaking in R?
Just to illustrate one example, for my Logistic Regression model, I used the following code in R:
set.seed(100) # use the same seed for every model
ftLogitC1 <- train(x = data2014c, y = class2014c, method = "glm",
metric = "ROC", preProcess = c("center", "scale"),
trControl = ctrl_C)
> ftLogitC1
Generalized Linear Model
513 samples
110 predictors
2 classes: 'Decrease', 'Increase'
Pre-processing: centered (110), scaled (110)
Resampling: Repeated Train/Test Splits Estimated (25 reps, 75%)
Summary of sample sizes: 386, 386, 386, 386, 386, 386, ...
Resampling results:
ROC Sens Spec
0.5706867 0.6142857 0.5017544
# compare the expected classifications in 2015 to the observed classifications in 2015
LogitC1_predictions_for_2015 <- predict(ftLogitC1, data2015c)
# create a confusion matrix to show how well our Logit model's predictions fared
# by inspecting all of its classification model performance metrics
LogitC1_CFM <- confusionMatrix(data = LogitC1_predictions_for_2015,
reference = class2015c, positive = "Increase")
And when I print out the above Confusion Matrix, I get an accuracy of only 0.395, a Kappa of -0.084, a True Positive Rate of 0.38, a True Negative Rate of 0.47, a Positive Predictive Value of 0.75, and an AUC of 0.613, which collectively is absolutely abysmal performance!
One important thing I should add here is that I have already asked an extremely similar question to this on Stack Overflow yesterday but have gotten no responses yet and I was not sure whether this sort of question would be more appropriate to ask here or there to be honest.
