Keras Model Produces Different Outputs, But Outputs Map To The Same Vector

Question

I have a neural network that produces output vectors from input vectors. These output vectors are different depending on what input vector it is being asked to predict for. I have a dictionary that maps these output vectors to human-readable values. Since the output vector is highly unlikely to be one of the vectors for which I have mappings, I am only trying to locate the closest vector.

However, no matter what I do, the output vector always maps to the same value in the dictionary. I have tried multiple different methods of comparing the two vectors, including calculating element-by-element, using KDTrees, and using vector magnitudes. Yet, every time, no matter the input to the network, the same "closest vector" is being found.

I'm stumped on this one. If all the outputs are different, you'd think they'd at least map to different vectors in the dictionary, even if they aren't the "right" ones. Any help is much appreciated.

    inputList = np.array(inputList)
outputList = np.array(outputList)

import tensorflow
from keras.models import Sequential
from keras.layers import Dense, Input
from tensorflow.keras.optimizers import Adam
import matplotlib.pyplot as plt
import numpy as np
import keras


# Create Keras model
model = Sequential()
model.add(Dense(1, input_dim=850))
model.add(Dense(25))
model.add(Dense(25))
model.add(Dense(25))
model.add(Dense(850))

# Gradient descent algorithm
adam = Adam(0.0001)

model.compile(loss='mse', optimizer=adam)
model.build((None, 850))
model.fit(inputList, outputList, epochs=1000, callbacks=tensorflow.keras.callbacks.EarlyStopping(monitor='loss', patience=3))
model.save(r'C:\Users\lasti\Downloads\keras3')
toSearch = np.fromstring(all["cat"], sep=" ")
toSearch = np.array([toSearch])
prediction1 = model.predict(toSearch)

all = {x: np.fromstring(all[x], sep=" ") for x in all}

def find_closest(x, array2d):
    least_diff = 999
    least_diff_index = -1
    for num, elm in enumerate(array2d):
        diff = [abs(x[count]-elm[count]) for count in range(850)]
        diff = sum(diff)
        if diff < least_diff:
            least_diff = diff
            least_diff_index = num
    return array2d[least_diff_index]
vals = np.fromiter(all.values(), dtype=object)
closest = find_closest(prediction1.T, vals)

for x, y in all.items():
   if(np.array_equal(y, closest)):
      print(x)

Note that inputList and outputList are just lists of numpy vectors. all maps from human-readable strings (the keys) to vectors (the values).

Answer 1

When you want to predict a discrete element (one of N possible outcomes), it's better not to use MSE, but to use categorical cross-entropy. For that, instead of using the vectors as expected outputs, use the indexes of the element in all. You can check any example of multiclass classification in Keras to have an example of how to do it in code.

Also, you should note that stacking multiple Dense layers is equivalent to a single Dense layer. You should include non-linearities (e.g. ReLU) in the intermediate layers. Check this answer for details on this issue.