0
$\begingroup$

I'm currently in the process of getting into data science, ML, and especially neural networks (coming from a "pure" software engineering background/degree). I did some models on classic datasets like MNIST and MNIST fashion that worked great and wanted to tackle a more difficult kind of problem next as a challenge: sports/soccer prediction. I've found some papers where this was already tried using deep neural networks with results of up to ~0.65 to 0.7 accuracy. This is a level I'd like to reach as well. However, I cannot get my model above around 0.507 accuracy. Here's what I did:

I have datasets of several thousand fixtures in leagues around the world (Premier League, Bundesliga, etc.) from the past 10 seasons. As input data, I'm using the following:

  • Weekday of the game
  • Season (year)
  • Round no
  • League id (one hot encoded)
  • Night match (yes/no)
  • Venue id (one hot encoded)
  • Home & Away team id (one hot encoded)
  • Home & Away team season points

As you can see, some of my data is one hot encoded. All the data points (including the data that is not one hot encoded) are placed into np arrays of the same size. So all the data is padded with zeroes until they are of the same size to guarantee the same input shape for every dataset of (10, 480). As I add more datasets, the last number grows bigger as there are more team ids, venue ids etc, so the needed array size/padding grows. I'm not entirely sure if this is the best solution, but as far as I understood, neural networks do depend on the input being the same shape, so this was the best solution in my eyes.

The model is then built like this:

def build_model(input_shape, dropout=0.2, hidden_layers=1, hidden_sizes=[64], learning_rate=0.01, optimizer="adam", last_layer_activation="softmax"):
  if optimizer == 'adam':
    optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
  elif optimizer == 'adagrad':
    optimizer = tf.keras.optimizers.Adagrad(learning_rate=learning_rate)
  elif optimizer == 'sgd':
    optimizer = tf.keras.optimizers.SGD(learning_rate=learning_rate)
  elif optimizer == 'rmsprop':
    optimizer = tf.keras.optimizers.RMSprop(learning_rate=learning_rate)
  elif optimizer == 'adadelta':
    optimizer = tf.keras.optimizers.Adadelta(learning_rate=learning_rate)
  elif optimizer == 'adamax':
    optimizer = tf.keras.optimizers.Adamax(learning_rate=learning_rate)
  else:
    raise ValueError(f"Unknown optimizer {optimizer}")

  model = tf.keras.models.Sequential()
  model.add(tf.keras.layers.Flatten(input_shape=input_shape))
  model.add(tf.keras.layers.Dropout(dropout))
  for i in range(hidden_layers):
    model.add(tf.keras.layers.Dense(hidden_sizes[i], activation='relu'))
  model.add(tf.keras.layers.Dense(3, activation=last_layer_activation))

  model.compile(optimizer=optimizer,
                loss='categorical_crossentropy',
                metrics=['accuracy'])
  return model

The data is then split up intro training & validation data with a 80:20 split and trained using model.fit(x_train, y_train, epochs=100, callbacks=callbacks, batch_size=64, validation_data=(x_val, y_val)) with an EarlyStop. I have also tried bigger batch sizes.

The model then outputs an one hot encoded array: [Home Win, Draw, Away Win].

I have tried tuning the model's hyperparameters (seen in the function parameters) with KerasTuner to no success. I cannot get above accuracy of about 0.507 max.

Also, my validation accuracy, as well as loss, is very noisy as you can see from this plot (where I've tried training for a little longer than 100 epochs): Training and Validation Accuracy - the validation accuracy converges at around 0.5 and is very noisy, the training accuracy keeps growing a little bit more and is less noisy

I would expect some noise (around the level of the training data) - due to the randomness factor of sports - but not this much noise. The training accuracy does seem to grow a little bit further which - if I understood correctly - speaks for overfitting which I've tried to reduce using Dropout layers.

Any ideas on how I could improve this model further? Is my data unfitting, e.g. should I include the lineup for each fixture? Too little data? Too much data? Is another network better fitted for the job?

Thanks!

Update 1: Based on similar problems, I've tried some things to rule some reasons out:

  • I've tried switching the training and validation datasets to see if there was some difference in preprocessing between the two. Same result.
  • I've also generated random datasets as the y values / one hot encoded results using np.eye(3)[np.random.choice(3, size=len(x_train))]. As expected, the model jumps around 0.33. So the model does indeed work, simply not very well.
$\endgroup$

1 Answer 1

0
$\begingroup$

I think you are plotting validation accuracy versus The number of steps. as step increases accuracy increases but at the end of the epoch the average accuracy is written which is less than the last step accuracy. That's why you will see a sharp decline in accuracy after each epoch but at the start of next epoch the accuracy will again go high.

  • there seems to be over fitting as training and validation accuracy are different so you can try to use regularisation parameter this will increase validation accuracy
  • plot training loss vs number of examples to check if you should increase the number of training examples
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.