0
$\begingroup$

Dataset

I have a dataset which is in the following format:

834 inputs: 2 floats between 0-1, and 832 integers created from one-hot-encoding 64 values (13 classes per value).

4096 outputs: Each is a one-hot-encoded class, such that the first number would be the first category, the second number would be the second category, and so on. There is only one correct output class, so only one value will be a 1.

Since of the 4096 output categories, not all of them are possible for each input, I've created a custom loss function which I pass into it two parts to the output.

  • 4096 output values. [0,0,0,0,...,1,0,0,0,0,...]
  • A list with 4096 integers of what outputs are possible. This is multiplied with the AI predictions since I don't care what the AI suggests for not-possible outputs. [0,0,1,0,0,...,0,1,1,0,0]

Model

The model is as follows:

def MakeModel():
    model = Sequential()
    model.add(Dense(256, input_dim=834, activation='relu'))
    model.add(BatchNormalization())
    for _ in range(2):
        model.add(Dense(256, activation='relu'))
        model.add(BatchNormalization())
    model.add(Dense(4096, activation='softmax'))
    model.compile(loss=customLoss, optimizer=Adam(amsgrad=True), metrics=['accuracy'])
    return model

Loss Function

The custom loss function is:

def customLoss(dataOut,aiOut):
    actualOut     = dataOut[:, 0:4096]
    possibleMoves = dataOut[:, 4096:8192]
    
    aiOutPossible = possibleMoves*aiOut     #This is the ai output, only including possible moves
    
    loss = tf.keras.backend.binary_crossentropy(actualOut, aiOutPossible)
    #loss = tf.keras.backend.categorical_crossentropy(actualOut, aiOutPossible)
    
    return loss

If I understand correctly, the correct loss function to use would be categorical_crossentropy, since I'm using 4096 one-hot-encoded outputs.

Results

Unfortunately, the results are very poor, with a validation accuracy of ~0.17%:

225/225 - 14s - loss: 6.0649 - accuracy: 0.0182 - val_loss: 4.2419 - val_accuracy: 0.0215
222/222 - 13s - loss: 3.7911 - accuracy: 0.0160 - val_loss: 3.2553 - val_accuracy: 0.0164
201/201 - 12s - loss: 3.0921 - accuracy: 0.0194 - val_loss: 2.9757 - val_accuracy: 0.0189
221/221 - 13s - loss: 2.9000 - accuracy: 0.0170 - val_loss: 2.8644 - val_accuracy: 0.0190
222/222 - 13s - loss: 2.8031 - accuracy: 0.0174 - val_loss: 2.8231 - val_accuracy: 0.0180
221/221 - 13s - loss: 2.7412 - accuracy: 0.0186 - val_loss: 2.7795 - val_accuracy: 0.0191
214/214 - 12s - loss: 2.7455 - accuracy: 0.0158 - val_loss: 2.7329 - val_accuracy: 0.0158
193/193 - 12s - loss: 2.6967 - accuracy: 0.0181 - val_loss: 2.7116 - val_accuracy: 0.0188
217/217 - 12s - loss: 2.6593 - accuracy: 0.0199 - val_loss: 2.6849 - val_accuracy: 0.0170
227/227 - 13s - loss: 2.6688 - accuracy: 0.0191 - val_loss: 2.6879 - val_accuracy: 0.0202
202/202 - 12s - loss: 2.6455 - accuracy: 0.0211 - val_loss: 2.6561 - val_accuracy: 0.0220
217/217 - 13s - loss: 2.6129 - accuracy: 0.0211 - val_loss: 2.6610 - val_accuracy: 0.0220
224/224 - 13s - loss: 2.6278 - accuracy: 0.0207 - val_loss: 2.6387 - val_accuracy: 0.0173
221/221 - 13s - loss: 2.5982 - accuracy: 0.0188 - val_loss: 2.6365 - val_accuracy: 0.0161
211/211 - 13s - loss: 2.5818 - accuracy: 0.0198 - val_loss: 2.6107 - val_accuracy: 0.0195
221/221 - 13s - loss: 2.6160 - accuracy: 0.0190 - val_loss: 2.6113 - val_accuracy: 0.0204
218/218 - 13s - loss: 2.5962 - accuracy: 0.0191 - val_loss: 2.5933 - val_accuracy: 0.0177
224/224 - 15s - loss: 2.5523 - accuracy: 0.0197 - val_loss: 2.5937 - val_accuracy: 0.0174
224/224 - 15s - loss: 2.5810 - accuracy: 0.0159 - val_loss: 2.5831 - val_accuracy: 0.0174
220/220 - 15s - loss: 2.5465 - accuracy: 0.0184 - val_loss: 2.5631 - val_accuracy: 0.0210
232/232 - 17s - loss: 2.5596 - accuracy: 0.0178 - val_loss: 2.5787 - val_accuracy: 0.0182
217/217 - 15s - loss: 2.5375 - accuracy: 0.0191 - val_loss: 2.5591 - val_accuracy: 0.0182
226/226 - 14s - loss: 2.5161 - accuracy: 0.0193 - val_loss: 2.5489 - val_accuracy: 0.0183
218/218 - 13s - loss: 2.5139 - accuracy: 0.0183 - val_loss: 2.5449 - val_accuracy: 0.0175
221/221 - 13s - loss: 2.4935 - accuracy: 0.0199 - val_loss: 2.5453 - val_accuracy: 0.0194
230/230 - 14s - loss: 2.5067 - accuracy: 0.0176 - val_loss: 2.5356 - val_accuracy: 0.0198
209/209 - 12s - loss: 2.4901 - accuracy: 0.0187 - val_loss: 2.5324 - val_accuracy: 0.0169
221/221 - 13s - loss: 2.4924 - accuracy: 0.0171 - val_loss: 2.5150 - val_accuracy: 0.0181
233/233 - 14s - loss: 2.4844 - accuracy: 0.0174 - val_loss: 2.5139 - val_accuracy: 0.0177
219/219 - 13s - loss: 2.4908 - accuracy: 0.0167 - val_loss: 2.5540 - val_accuracy: 0.0167
212/212 - 13s - loss: 2.4907 - accuracy: 0.0191 - val_loss: 2.5199 - val_accuracy: 0.0190
227/227 - 13s - loss: 2.4772 - accuracy: 0.0160 - val_loss: 2.5036 - val_accuracy: 0.0175
226/226 - 14s - loss: 2.4818 - accuracy: 0.0169 - val_loss: 2.5041 - val_accuracy: 0.0177
219/219 - 13s - loss: 2.4773 - accuracy: 0.0168 - val_loss: 2.4996 - val_accuracy: 0.0157
217/217 - 13s - loss: 2.4706 - accuracy: 0.0171 - val_loss: 2.5008 - val_accuracy: 0.0175
228/228 - 14s - loss: 2.4687 - accuracy: 0.0167 - val_loss: 2.4900 - val_accuracy: 0.0170
222/222 - 13s - loss: 2.4495 - accuracy: 0.0174 - val_loss: 2.4872 - val_accuracy: 0.0156
219/219 - 13s - loss: 2.4442 - accuracy: 0.0167 - val_loss: 2.4824 - val_accuracy: 0.0152
217/217 - 13s - loss: 2.4519 - accuracy: 0.0162 - val_loss: 2.4799 - val_accuracy: 0.0170
218/218 - 14s - loss: 2.4606 - accuracy: 0.0147 - val_loss: 2.4775 - val_accuracy: 0.0170
220/220 - 14s - loss: 2.4382 - accuracy: 0.0173 - val_loss: 2.4724 - val_accuracy: 0.0145
209/209 - 13s - loss: 2.4238 - accuracy: 0.0170 - val_loss: 2.4657 - val_accuracy: 0.0154
212/212 - 13s - loss: 2.4480 - accuracy: 0.0148 - val_loss: 2.4657 - val_accuracy: 0.0140
226/226 - 14s - loss: 2.4373 - accuracy: 0.0157 - val_loss: 2.4677 - val_accuracy: 0.0177
231/231 - 14s - loss: 2.4427 - accuracy: 0.0152 - val_loss: 2.4690 - val_accuracy: 0.0155
216/216 - 13s - loss: 2.4252 - accuracy: 0.0157 - val_loss: 2.4670 - val_accuracy: 0.0165
225/225 - 15s - loss: 2.4437 - accuracy: 0.0147 - val_loss: 2.4518 - val_accuracy: 0.0148
226/226 - 16s - loss: 2.4178 - accuracy: 0.0144 - val_loss: 2.4503 - val_accuracy: 0.0158
235/235 - 15s - loss: 2.4281 - accuracy: 0.0146 - val_loss: 2.4495 - val_accuracy: 0.0142
218/218 - 15s - loss: 2.4193 - accuracy: 0.0144 - val_loss: 2.4502 - val_accuracy: 0.0137
216/216 - 15s - loss: 2.4175 - accuracy: 0.0144 - val_loss: 2.4530 - val_accuracy: 0.0149
232/232 - 14s - loss: 2.4210 - accuracy: 0.0142 - val_loss: 2.4441 - val_accuracy: 0.0145
226/226 - 14s - loss: 2.4304 - accuracy: 0.0140 - val_loss: 2.4549 - val_accuracy: 0.0160
219/219 - 13s - loss: 2.4300 - accuracy: 0.0165 - val_loss: 2.4584 - val_accuracy: 0.0163
223/223 - 14s - loss: 2.4165 - accuracy: 0.0146 - val_loss: 2.4426 - val_accuracy: 0.0152
217/217 - 14s - loss: 2.4247 - accuracy: 0.0150 - val_loss: 2.4391 - val_accuracy: 0.0144
216/216 - 14s - loss: 2.4271 - accuracy: 0.0146 - val_loss: 2.4360 - val_accuracy: 0.0156

However, when trying the same problem with binary_crossentropy:

225/225 - 25s - loss: 0.0018 - accuracy: 0.0348 - val_loss: 0.0017 - val_accuracy: 0.0383
222/222 - 24s - loss: 0.0016 - accuracy: 0.0624 - val_loss: 0.0015 - val_accuracy: 0.0633
201/201 - 22s - loss: 0.0014 - accuracy: 0.0778 - val_loss: 0.0014 - val_accuracy: 0.0805
221/221 - 23s - loss: 0.0013 - accuracy: 0.0830 - val_loss: 0.0013 - val_accuracy: 0.0929
222/222 - 23s - loss: 0.0013 - accuracy: 0.0968 - val_loss: 0.0013 - val_accuracy: 0.0933
221/221 - 23s - loss: 0.0013 - accuracy: 0.1051 - val_loss: 0.0013 - val_accuracy: 0.1024
214/214 - 29s - loss: 0.0012 - accuracy: 0.1001 - val_loss: 0.0012 - val_accuracy: 0.1007
193/193 - 21s - loss: 0.0012 - accuracy: 0.1060 - val_loss: 0.0012 - val_accuracy: 0.1079
217/217 - 23s - loss: 0.0012 - accuracy: 0.1165 - val_loss: 0.0012 - val_accuracy: 0.1156
227/227 - 25s - loss: 0.0012 - accuracy: 0.1172 - val_loss: 0.0012 - val_accuracy: 0.1183
202/202 - 21s - loss: 0.0012 - accuracy: 0.1177 - val_loss: 0.0012 - val_accuracy: 0.1176
217/217 - 25s - loss: 0.0011 - accuracy: 0.1308 - val_loss: 0.0012 - val_accuracy: 0.1180
224/224 - 24s - loss: 0.0012 - accuracy: 0.1238 - val_loss: 0.0011 - val_accuracy: 0.1273
221/221 - 20s - loss: 0.0011 - accuracy: 0.1209 - val_loss: 0.0011 - val_accuracy: 0.1253
211/211 - 24s - loss: 0.0011 - accuracy: 0.1285 - val_loss: 0.0011 - val_accuracy: 0.1278
221/221 - 23s - loss: 0.0011 - accuracy: 0.1230 - val_loss: 0.0011 - val_accuracy: 0.1159
218/218 - 23s - loss: 0.0011 - accuracy: 0.1308 - val_loss: 0.0011 - val_accuracy: 0.1325
224/224 - 24s - loss: 0.0011 - accuracy: 0.1333 - val_loss: 0.0011 - val_accuracy: 0.1343
224/224 - 21s - loss: 0.0011 - accuracy: 0.1249 - val_loss: 0.0011 - val_accuracy: 0.1305
220/220 - 21s - loss: 0.0011 - accuracy: 0.1359 - val_loss: 0.0011 - val_accuracy: 0.1371
232/232 - 22s - loss: 0.0011 - accuracy: 0.1318 - val_loss: 0.0011 - val_accuracy: 0.1369
217/217 - 21s - loss: 0.0011 - accuracy: 0.1384 - val_loss: 0.0011 - val_accuracy: 0.1361
226/226 - 21s - loss: 0.0011 - accuracy: 0.1357 - val_loss: 0.0011 - val_accuracy: 0.1353
218/218 - 21s - loss: 0.0011 - accuracy: 0.1386 - val_loss: 0.0011 - val_accuracy: 0.1398
221/221 - 23s - loss: 0.0011 - accuracy: 0.1439 - val_loss: 0.0011 - val_accuracy: 0.1412
230/230 - 25s - loss: 0.0011 - accuracy: 0.1391 - val_loss: 0.0011 - val_accuracy: 0.1383
209/209 - 21s - loss: 0.0011 - accuracy: 0.1454 - val_loss: 0.0011 - val_accuracy: 0.1395
221/221 - 20s - loss: 0.0011 - accuracy: 0.1462 - val_loss: 0.0011 - val_accuracy: 0.1452
233/233 - 24s - loss: 0.0011 - accuracy: 0.1443 - val_loss: 0.0011 - val_accuracy: 0.1422
219/219 - 23s - loss: 0.0011 - accuracy: 0.1458 - val_loss: 0.0011 - val_accuracy: 0.1420
212/212 - 23s - loss: 0.0011 - accuracy: 0.1471 - val_loss: 0.0011 - val_accuracy: 0.1477
227/227 - 27s - loss: 0.0011 - accuracy: 0.1438 - val_loss: 0.0011 - val_accuracy: 0.1479
226/226 - 22s - loss: 0.0011 - accuracy: 0.1457 - val_loss: 0.0011 - val_accuracy: 0.1443
219/219 - 20s - loss: 0.0011 - accuracy: 0.1499 - val_loss: 0.0011 - val_accuracy: 0.1481
217/217 - 23s - loss: 0.0011 - accuracy: 0.1502 - val_loss: 0.0011 - val_accuracy: 0.1460
228/228 - 22s - loss: 0.0011 - accuracy: 0.1495 - val_loss: 0.0011 - val_accuracy: 0.1458
222/222 - 23s - loss: 0.0011 - accuracy: 0.1538 - val_loss: 0.0011 - val_accuracy: 0.1516
219/219 - 25s - loss: 0.0011 - accuracy: 0.1553 - val_loss: 0.0011 - val_accuracy: 0.1551
217/217 - 21s - loss: 0.0011 - accuracy: 0.1535 - val_loss: 0.0011 - val_accuracy: 0.1525

Here's a nice graph showing the accuracy using binary_crossentropy over more epochs: Binary classification accuracy

So my main questions are:

  1. What is the correct classification to use?
  2. Why is my validation accuracy so low, and what can I do to improve it?
Improve this question
$\endgroup$
3
  • $\begingroup$ Are you sure your results are so poor? Assuming balanced classes, you go into the problem with only a $\sim0.024\%$ chance of a correct classification. You are beating that by about a factor of $8$. $\endgroup$
    – Dave
    Aug 30, 2021 at 14:16
  • $\begingroup$ @Dave From what other people have created, ~70% accuracy was achieved. Compared to what I'm getting, the NN's got a long way to go. $\endgroup$
    – Rlz
    Aug 30, 2021 at 14:19
  • $\begingroup$ Still stuck on this issue. Perhaps it's due to stats.stackexchange.com/questions/285255/…? Check comments on the answer. $\endgroup$
    – Rlz
    Sep 22, 2021 at 9:56

1 Answer 1

Reset to default
0
$\begingroup$

If the classes are mutually exclusive then ideally you should use categorical cross entropy for the loss function. Binary loss should still work after a fashion though, since it will still encourage incorrect classes to predict low probability, and correct classes high.

You do not appear to be renormalising the value of aiOutPossible before using categorical_crossentropy. You should definitely do so, because without that step, your class probabilities will not sum to 1, which will skew calculations significantly.

I think something like:

aiOutPossible = aiOutPossible / aiOutPossible.sum()

before calling tf.keras.backend.categorical_crossentropy should do the trick in your custom loss function.

You may also find that you don't really need this custom loss function during training - the extra benefit of focussing on only relevant outputs may be offset by not penalising high/confident scores for impossible outputs. So you may also consider training without the filter and custom loss, only using the filter when scoring test data and in production. This will allow you to use some built-in gradient calcutions in TensforFlow which are more numerically stable (use from_logits=true). I cannot say whether this will actually help with any confidence, but it is worth checking IMO as it will simplify your training process.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the response! I've just tested it with renormalisation (although I couldn't use .sum() since Tensors don't have it apparently). The code & results are here: pastebin.com/WjUWFpBs. As for your second point, typically only 0.7% of the outputs are valid, the other 99.3% are invalid and are what I'm attempting to ignore in the custom loss function. With this in mind, do you think it's still worth a try to penalise it for scoring impossible outputs highly? $\endgroup$
    – Rlz
    Aug 30, 2021 at 15:48
  • $\begingroup$ @RulerOfTheWorld That is a low percentage, so maybe my suggestion is not helpful. The network is calculating all the values anyway though, so it may still be worth the experiment. If there is some way to overlap and re-use categories that makes sense in your problem domain, that may also help. For that to make sense, there would need to be similar classes that were never available at the same time, that you could count as one category interpretted according to the current filter. $\endgroup$
    – Neil Slater
    Aug 30, 2021 at 16:01
  • 1
    $\begingroup$ So if I understand you correctly, the filters should reduce the possibilties down from 1 in 4096 to around 1 in 30? Therefore an accuracy of 0.03 would be the same as random guessing. You are posting starting accuracies around 0.001 and final accuracies under 0.02. So it is likely you have an implementation error. I cannot see anything though $\endgroup$
    – Neil Slater
    Aug 30, 2021 at 16:06
  • $\begingroup$ That does make sense, it should definitely be higher than what I'm getting. If I remove the multiplication in the custom loss function, binary_crossentropy is around 0.039 after 12 epochs, and categorical_crossentropy is around 0.1285 after 9 epochs. Full output at: pastebin.com/LBPvGyJ8 $\endgroup$
    – Rlz
    Aug 30, 2021 at 16:21
  • $\begingroup$ Some very strange results here, and I have no idea what's causing it... The custom loss function, without normalisation, with binary_crossentropy gives almost the exact same output as using the default categorical_crossentropy (not in the custom loss function). If I do any other combination (custom loss + normalisation + categorical_crossentropy) the accuracy drops to almost zero. Which seems completely counter-intuitive, since the custom loss function removes 99.7% of the wrong answers for the network which should improve the results, but it somehow makes it worse. $\endgroup$
    – Rlz
    Aug 30, 2021 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.