Autoencoder general questions and poor loss

Question

I'm trying to get a simple autoencoder working on the iris dataset to explore autoencoders at a basic level. I'm running into an issue where the loss of the model is extremely high (>20).

Can someone help me understand if this model looks normal to them to begin with?

Some questions I'd love some help on understanding:

• There are 3 possible outputs for y - thus I used Softmax in the final layer - if I was to OHE the output, would using something like Sigmoid be more appropriate as the values are bound between 0 and 1?

• Altering the smallest change in the layers (encoding layer going to 6 instead of 3) --> causes a major shift in the loss -- is this normal?

• Each run of the autoencoder produces a different result - is this normal that it is not deterministic?

• Why does the last layer have to be the same size (4) as the input dimension - are we able to force this to allow for an output of 3 for example? I know I can read from a latent layer, but then I can't fit the model based on that layer.

    import pandas as pd
    import numpy as np
  
    from sklearn.model_selection import cross_val_score, train_test_split
    from sklearn.preprocessing import LabelEncoder, OneHotEncoder
    from sklearn.pipeline import Pipeline
    from sklearn import datasets
    from tensorflow.keras.layers import Input, Dense, BatchNormalization, LeakyReLU
    from tensorflow.keras import backend, layers, models, metrics, utils
    from tensorflow.keras import regularizers, Input, Model, optimizers
  
  
    iris = datasets.load_iris()
    x = iris.data
    y = iris.target.reshape(-1, 1)
    X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.20)
    input_dim = Input(shape=(X_train.shape[1],))
  
  
    encoded = layers.Dense(6, input_dim='input_dim')(input_dim)
    encoded = BatchNormalization()(encoded)
    encoded = LeakyReLU()(encoded)
    encoded = layers.Dense(3)(encoded)
  
    decoded = layers.Dense(4, activation='softmax')(encoded)
    autoencoder = Model(inputs=input_dim, outputs=decoded)
  
  
    opt = optimizers.Adam(lr=0.00001)
    autoencoder.compile(optimizer=opt
                        , loss='categorical_crossentropy'
                        , metrics=[metrics.CategoricalAccuracy()])
  
  
    history = autoencoder.fit([X_train]
     , [X_train]
     , epochs=16
     , batch_size=2
     , verbose=2
     , validation_data=((X_test),(X_test))
                       )
  

Thank you for any help!

Answer 1

There is some confusion in your question, since you are talking about an AE, but you are using softmax as output... which makes almost no sense

First of all, the loss by itself, has no meaning:
If I give you two models with two different losses (in your case, two AE), you cannot, in any way, guess which is the best one (therefore a loss that is 20 means nothing)

About your questions:

1. depends on the distribution... one hot encoding an output supposes a multinomial distribution, which you can approximate with both a softmax and sigmoid, but 99% of the times you should use a softmax, since encodes constraints of the distribution directly, and the NN don't have to learn them (like when approximating a probability, you can for sure use a linear output layer, but the NN has to learn to output values between 0 and 1, and there is no guarantee that it will always do that)
2. an AE aims to reduce the dimensionality... you start with 4 features and you allow it to use 6 dimensions (in this case we are talking about overcomplete AE, but you need some constraint to make them work as expected)
3. welcome in the world of non convex optimization... if the starting point is random (which is since the weights are initialized randomly), you will probably get to a different local minima
4. an autoencoder aims to learn the identity function ($decoder(encoder(x)) = x$) therefore obviously no, the input has to have the same shape as the output

At this point, I'm pretty certain that an AE is not what you are looking for, but most likely a normal discriminative NN (since you are using cat-cross-ent as loss function)