1
$\begingroup$

As a personal project, I'm trying to build a classifier which attempts to predict the metacritic score of a game based purely on its cover. I figured it would be a fun project to learn Keras image classification with (2D-Convolution to be more specific).

I'm definitely new to this so if I've made any rookie mistakes, please tell me. Here are some notes before I post the CNN:

1) I've written a Metacritic Scrubber which takes PS2, PS3, PS4, Xbox360 and Xbox One games which have metacritic scores, downloads the artwork and labels it with the score. Duplicate games are removed (which may be a mistake since they sometimes have different scores based on platform). I'm pretty happy with this code.

2) I've rounded scores on a scale of 0-9 instead of 0-100 to make the number of classifications smaller.

3) images are 123x98 with 3 channels. All images have been stretched to this size. I wonder if this may be a source of problems because some covers have been stretched. Values are between 0-255 for each channel.

4) Games with square cover art (DLC, non-retail, etc) have been omitted.

This leaves me with a data set of 3816 game covers. I figured this would probably be enough for an initial investigation.

The model I've built has been based on the work by Iwana et al in this paper: https://arxiv.org/abs/1610.09204

model = Sequential()
model.add(Conv2D(64, (2, 3),activation='relu', input_shape=(123,98,3)))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(128, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(256, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(256, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Conv2D(256, (2, 3),activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='valid'))

model.add(Flatten())

model.add(Dense(720, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(720, activation='relu'))
model.add(Dense(720, activation='relu'))
model.add(Dense(10, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.001,momentum=0.1,decay=0.0005,nesterov=True), metrics=['accuracy'])
model.summary()

#Save output to file
csv_logger = CSVLogger('training.log')
#Save best model when possible
checkpoint = ModelCheckpoint('training_model_best.hdf5', monitor='val_acc', verbose=1, save_best_only=True, mode='max')

model.fit(np.array(images), np.array(labels), epochs=225, batch_size=10,
    validation_split=0.50,verbose=2,shuffle=True,callbacks=[csv_logger,checkpoint])

I'm using quite a harsh validation split of 0.5 because I really wanted to avoid overfitting. But regardless of whether I use between 0.2 - 0.5, I typically reach a validation accuracy of approximately 35%, which is pretty disappointing. A lot of the hyper-parameters I have no idea what to set to however.

Sorry for the long post. Ultimately, my result is a little better than random guessing but I'm hoping through some pointers, I can bring this accuracy up a bit. I wonder if actually, the cover files I've downloaded are too small.

Thanks for reading.

$\endgroup$
1
$\begingroup$

Here are some points to consider:

1) I've written a Metacritic Scrubber which takes PS2, PS3, PS4, Xbox360 and Xbox One games which have metacritic scores, downloads the artwork and labels it with the score. Duplicate games are removed (which may be a mistake since they sometimes have different scores based on platform). I'm pretty happy with this code.

Good!

2) I've rounded scores on a scale of 0-9 instead of 0-100 to make the number of classifications smaller.

This also should be fine.

3) images are 123x98 with 3 channels. All images have been stretched to this size. I wonder if this may be a source of problems because some covers have been stretched. Values are between 0-255 for each channel.

I would normalise the values between 0-1 (norm_values = values/255). Regarding the size of the files I don't think this should be a problem. For example, Cifar-10 are 32x32 or the average image resolution on ImageNet is 469x387 pixels although most approaches resize them to 256x256. I think your size should be fine.

4) Games with square cover art (DLC, non-retail, etc) have been omitted.

Not much to add here.

This leaves me with a data set of 3816 game covers. I figured this would probably be enough for an initial investigation. The model I've built has been based on the work by Iwana et al in this paper: https://arxiv.org/abs/1610.09204

Here is where I actually think we have some problems. First, your dataset is quite small. 3816 reduced to 1908 when splitting train/test... This is not good especially with such structure. In the paper you mentioned, they are using a network with around 2.5M parameters and they used ~137K samples. Your 1908 dataset seems tiny compared and you are using the same model structure...

In my opinion your model is not able to do better, simply put. It doesn't matter what parameters you choose you need (a lot) more data. You might try reducing the size of your network, also create more data using noise addition, mirror samples, etc... and see if these help somehow.

Finally, apart from the size, you have no way to tell if your data is representative enough so that any model can learn from it. Therefore, 35% accuracy is as good as any other value I am afraid. Anyway, your mission seemed to be able to complete your experiment and you did. And you learnt a lot about limitations of deep models, so I'd say: good work!

$\endgroup$
  • $\begingroup$ Hey thanks for the reply. It's comforting to know I haven't done anything that wrong. The data augmentation is a really good idea which I hadn't really considered for this problem. I'm skeptical of adder systems older than the chosen ones because I have a feeling artistic trends vary too much over the decade. I very much want to reach a point where I can investigate what the model has learnt. I could perhaps identify games it is very certain of their scores and see why it did that. Thanks again for the reply. $\endgroup$ – user1147964 Mar 9 '18 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.