1
$\begingroup$

I am trying to explore models for predicting whether the a team will win or lose based on features about the team and their opponent. My training data is 15k samples with 760 numerical features. Each sample represents a game between two teams and the features are long and short term statistics about each team at the time of the game (i.e. avg points over last 10 games).

My thought was to use a binary classifier as a multi layered perceptron. Each layer has batch norm, dropout and an relu activation. At the output layer there is a sigmoid activation. I am also using principal component analysis to reduce the dimensionality of my dataset.

Im am using an automated hyper-parameter tuner with randomized search and population based training to find optimal accuracy hyper-parameters. I am able to tune the hyper-parameters such that training loss and validation loss both converge to a very small value but when I look at accuracy I am getting very minimal improvements.

When looking at the initial accuracy after one epoch of training, it seems to be around 60% across all initial hyperparams, and it only reaches a max of about 70% after 30 epochs.

Im not really sure what to do, does this imply my data is to noisy / random? Should I reconsider my architecture? Would I get better results with a regression instead of classification? Also, since the data is sequential in nature this might lend itself to a recurrent architecture but Im not familiar with any RNN that is trained on multiple sequences as opposed to a single sequence.

$\endgroup$
0
$\begingroup$

It is hard to tell if it is the model or the signal/noise ratio in data. One sanity check could be to see how well a human would perform with the most important features.

I don't think you want to switch to a sequence model, I have a hard time seeing it do better for this data. Also, don't switch to regression.

Some suggestions:

  • Try a gradient boosting model, like LightGBM or XGBoost. It is really good on tabular data.
  • Do feature engineering and try to come up with features that might be good but does not exist in the data. You could aggregate to get the sequential information in there, but you already seem to be doing that to some extent.
|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks, I currently have 1 game, 5 game, and 10 game aggregates for each of the teams stats as a feature so each sample includes some sequential information. Could I use PCA as my feature engineering set or should I try to do feature engineering by hand? $\endgroup$ – dadrake Mar 11 '19 at 20:26
  • $\begingroup$ PCA does not really do any feature engineering at all. It actually does the opposite, it takes away information. But it is necessary to avoid training for days. So feature engineering by hand is good, especially if you try gradient boosting. $\endgroup$ – Simon Larsson Mar 11 '19 at 20:31
  • $\begingroup$ So would you then recommend to include PCA after manual feature engineering? OR just forgo PCA ? $\endgroup$ – dadrake Mar 11 '19 at 20:44
  • $\begingroup$ If you do gradient boosting you can train without PCA, it should be fast enough. $\endgroup$ – Simon Larsson Mar 11 '19 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.