# My network doesnt learn how to memorize noise

im trying to implement an idea i have and it involves letting the NN memorise a noise to image mapping of the cifar100 dataset. It uses a custom layer ....but even when i replace the custom layer with standard dense ones (in keras) the network doesnt seem to memorise the relationships,(the loss and acc improve slightly and then get stuck ) so im guessing its not the custom layer thats in the way...any thoughts?

examples=50000
n_nodes = 256 * 16 * 16
random_dim = 100
input_tensor = Input(shape=(random_dim,))
noise = np.random.randn(examples, random_dim)

(x_train, y_train), (x_test, y_test) = cifar100.load_data()
model = Sequential()

#MyLayer()
#MyLayer()

model.fit(noise, x_train,
epochs=250,
batch_size=212,
shuffle=True)



To start with, your architecture looks awkward; binary cross-entropy loss with accuracy metric are supposed to be used for classification problems, where the final layer needs to be a single-unit dense one with activation='sigmoid'. It is not at all clear what your network does here, with the chosen loss function and a convolutional final layer.

Second, since you don't define any activation function, all your layers use the default linear one. Recall from the Keras documentation the Dense layer:

keras.layers.Dense(units, activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)


where

activation: Activation function to use (see activations). If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).

Since you don't specify explicitly any activation, you actually use a linear one for all your layers. And it is well-known that a neural network comprised simply of linear units is equivalent with a simple linear unit (check Andrew Ng's lecture Why Non-linear Activation Functions for a detailed explanation); in fact, it is only with non-linear activation functions that neural networks begin to be able to do interesting things.

Third, the input_dim argument is supposed to be used only for the first layer (and not for intermediate ones, as you have done here); in intermediate layers, the input dimension is implicitly calculated as the number of outputs of the previous layer. See the SO thread Keras Sequential model input layer for more details.

• Thanks for the feedback, i added relu activations after each layer, but my network still performs exactly the same. its not learning or memorising anything past a point.My loss is stuck at 19302.5799 – Tofara Moyo Sep 14 '19 at 11:54
• @TofaraMoyo as I already said, your architecture & setting both look quite awkward, and it does not look like a well-posed problem. You may want to try autoencoders... – desertnaut Sep 14 '19 at 12:02
• I did get the code from an example of an autoencoder, i want to map a random noise vector to an image, rather than an image back on itself, this should memorise at least even if it cannot generelise to generating new images with new noise vectors.i have a custom layer built to handle the process of generalisation, but its not learning , so i tried a normal dense layer to see if it would memorise that ,to know wether it was the custom layer or not. i am implementing this paper researchgate.net/publication/… – Tofara Moyo Sep 14 '19 at 12:10