1
$\begingroup$

I am trying to explore models for predicting whether a team will win or lose based on features about the team and their opponent. My training data is 15k samples with 760 numerical features. Each sample represents a game between two teams and the features are long and short term statistics about each team at the time of the game (i.e. avg points over last 10 games).

My thought was to use a binary classifier as a multi layered perceptron. Each layer has batch norm, dropout, and an ReLU activation. At the output layer there is a sigmoid activation. I am also using principal component analysis to reduce the dimensionality of my dataset.

I am using an automated hyper-parameter tuner with randomized search and population based training to find optimal accuracy of the hyper-parameters. I am able to tune the hyper-parameters such that training loss and validation loss both converge to a very small value, but when I look at accuracy I am getting very minimal improvements.

When looking at the initial accuracy after one epoch of training, it seems to be around 60% across all initial hyper-parameters, and it only reaches a max of about 70% after 30 epochs.

I am not really sure what to do. Does this imply my data is to noisy/random? Should I reconsider my architecture? Would I get better results with a regression instead of classification? Also, since the data is sequential in nature this might lend itself to a recurrent architecture, but I am not familiar with any RNN that is trained on multiple sequences as opposed to a single sequence.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

It is hard to tell if it is the model or the signal/noise ratio in data. One sanity check could be to see how well a human would perform with the most important features.

I do not think you want to switch to a sequence model. I have a hard time seeing that it would do better for this data. Also, do not switch to regression.

Some suggestions:

  • Try a gradient boosting model, like LightGBM or XGBoost. It is really good on tabular data.
  • Do feature engineering and try to come up with features that might be good but do not exist in the data. You could aggregate to get the sequential information in there, but you already seem to be doing that to some extent.
Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks, I currently have 1 game, 5 game, and 10 game aggregates for each of the teams stats as a feature so each sample includes some sequential information. Could I use PCA as my feature engineering set or should I try to do feature engineering by hand? $\endgroup$
    – dadrake
    Commented Mar 11, 2019 at 20:26
  • $\begingroup$ PCA does not really do any feature engineering at all. It actually does the opposite, it takes away information. But it is necessary to avoid training for days. So feature engineering by hand is good, especially if you try gradient boosting. $\endgroup$
    – Simon Larsson
    Commented Mar 11, 2019 at 20:31
  • $\begingroup$ So would you then recommend to include PCA after manual feature engineering? OR just forgo PCA ? $\endgroup$
    – dadrake
    Commented Mar 11, 2019 at 20:44
  • $\begingroup$ If you do gradient boosting you can train without PCA, it should be fast enough. $\endgroup$
    – Simon Larsson
    Commented Mar 11, 2019 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.