2
$\begingroup$

This case has an underlying story but I have essentially boiled it down to the simplest possible re-producible example I could.

Essentially let us think that I have up to 1000 nodes and each node represented by a small (this case is a 3-cell vector) vector and I concatenate and represent these nodes as a padded 3*1000 input vector and need to find out which one is more suitable. So the model is trying to predict 1000 float values, one for each node.

Let's imagine the function to score nodes is this arbitrary code:

def score_vector(v):
  a, b, c = tuple(v)
  if a == 0 or b == 0 or a - c < 2:
    return float(Defs.INVALID_SCORE)
  return float(a * math.sqrt(a - c) / math.log(b + 2, 5))

And essentially my model is supposed to learn this function plus an argmax to find the node that has the highest score. This looks to me like a pretty simple problem compared to the problems I have solved so far (but it is different too).

So my question is why doesn't this model converge? I am thinking it could be due to its differentiability but really kinda lost and started to doubt everything I know about NN (which is not a lot).

Here is the repro code:

import numpy as np
import math

from keras import Sequential, Input
from keras.layers import Flatten, Activation, Dense
from keras.optimizers import Adam

class Defs:
  VECTOR_SIZE=3
  NODE_COUNT=1000
  MAX_REAL_NODE_COUNT=400
  MIN_REAL_NODE_COUNT=20
  INVALID_SCORE=0

def score_vector(v):
  a, b, c = tuple(v)
  if a == 0 or b == 0 or a - c < 2:
    return float(Defs.INVALID_SCORE)
  return float(a * math.sqrt(a - c) / math.log(b + 2, 5))

def build_vector():
  a = np.random.randint(1, 100)
  c = np.random.randint(1, 50) if np.random.choice([False, True, True]) else 0
  b = 0 if c == 0 else np.random.randint(c, c*3)
  return [float(a), float(b), float(c)]

def build_vectorset_score():
  n = np.random.randint(Defs.MIN_REAL_NODE_COUNT, Defs.MAX_REAL_NODE_COUNT)
  vectorset = []
  for i in range(0, n):
    vectorset += build_vector()

  # pad it
  vectorset += [0. for i in range((Defs.NODE_COUNT-n) * Defs.VECTOR_SIZE)]
  scores = [score_vector(vectorset[i*Defs.VECTOR_SIZE:(i+1)*Defs.VECTOR_SIZE]) for i in range(0, Defs.NODE_COUNT)]
  index = np.argmax(scores)
  scores = [1. if index == i else 0. for i in range(0, len(scores))]
  return vectorset, scores

def build_model():
  model = Sequential()
  model.add(Dense(Defs.VECTOR_SIZE * Defs.NODE_COUNT, input_dim=Defs.VECTOR_SIZE * Defs.NODE_COUNT, activation='relu'))
  model.add(Dense(Defs.NODE_COUNT, activation='relu'))
  model.add(Dense(Defs.NODE_COUNT))
  model.add(Activation('softmax'))
  print(model.summary())
  model.compile(loss="categorical_crossentropy",
                optimizer=Adam(lr=0.001), metrics=['categorical_accuracy'])
  return model


if __name__ == '__main__':
  SAMPLE_SIZE = 1 * 1000
  X = []
  Y = []
  for i in range(0, SAMPLE_SIZE):
    x, y = build_vectorset_score()
    X.append(np.array(x))
    Y.append(np.array(y))
  model = build_model()
  model.fit(np.array(X),
                  np.array(Y), batch_size=100, epochs=200, verbose=1)
$\endgroup$
3
$\begingroup$

I'm not totally sure exactly what you're doing with your scoring equation, but the first thing you need to look at is your loss function. Categorical Crossentropy is for multilabel classification, and you're trying to predict a float value.

So, you should have your network output be a single value (and don't squash it through a sigmoid unless the range of your function is (0,1)). You should be using a regression loss function - I'd definitely start with mean squared error. Check out the regression example here (under Regression Predictions) for some sample code.

Edit: Further to discussion below, the network cannot predict the index of the highest-valued node because each node's score has no dependency on the node's position among the 1000 nodes being scored. As the probability of the best node being in each position is the same, there is no optimum prediction to be learned.

| improve this answer | |
$\endgroup$
  • $\begingroup$ +1. Thank you. But as I explained, the idea is to have a network that scores 1000 nodes and selects the best one. I am not trying to predict a float, I am trying to predict 1000 float values. $\endgroup$ – Aliostad Aug 9 '18 at 18:58
  • $\begingroup$ OK - I misunderstood that part of the question, sorry about that. I still misunderstand part of what you're doing: why should you be able to predict the index of the highest-valued vector? There's no time-series-like dependence among the nodes, correct? Meaning that any permutation of the order of the nodes would be equally valid as training data (alternately, that the probability of the best node being in any one of the 1000 positions is the same)? If the best one could be anywhere, then the network correctly doesn't converge because no prediction is better than any other. $\endgroup$ – Matthew Aug 9 '18 at 19:13
  • $\begingroup$ Yes, permutation makes sense. The problem is that, happy to accept if you could change the answer to include this. $\endgroup$ – Aliostad Aug 9 '18 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.