0
$\begingroup$

I'm trying to get a simple autoencoder working on the iris dataset to explore autoencoders at a basic level. I'm running into an issue where the loss of the model is extremely high (>20).

Can someone help me understand if this model looks normal to them to begin with?

Some questions I'd love some help on understanding:

  • There are 3 possible outputs for y - thus I used Softmax in the final layer - if I was to OHE the output, would using something like Sigmoid be more appropriate as the values are bound between 0 and 1?

  • Altering the smallest change in the layers (encoding layer going to 6 instead of 3) --> causes a major shift in the loss -- is this normal?

  • Each run of the autoencoder produces a different result - is this normal that it is not deterministic?

  • Why does the last layer have to be the same size (4) as the input dimension - are we able to force this to allow for an output of 3 for example? I know I can read from a latent layer, but then I can't fit the model based on that layer.

      import pandas as pd
      import numpy as np
    
      from sklearn.model_selection import cross_val_score, train_test_split
      from sklearn.preprocessing import LabelEncoder, OneHotEncoder
      from sklearn.pipeline import Pipeline
      from sklearn import datasets
      from tensorflow.keras.layers import Input, Dense, BatchNormalization, LeakyReLU
      from tensorflow.keras import backend, layers, models, metrics, utils
      from tensorflow.keras import regularizers, Input, Model, optimizers
    
    
      iris = datasets.load_iris()
      x = iris.data
      y = iris.target.reshape(-1, 1)
      X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.20)
      input_dim = Input(shape=(X_train.shape[1],))
    
    
      encoded = layers.Dense(6, input_dim='input_dim')(input_dim)
      encoded = BatchNormalization()(encoded)
      encoded = LeakyReLU()(encoded)
      encoded = layers.Dense(3)(encoded)
    
      decoded = layers.Dense(4, activation='softmax')(encoded)
      autoencoder = Model(inputs=input_dim, outputs=decoded)
    
    
      opt = optimizers.Adam(lr=0.00001)
      autoencoder.compile(optimizer=opt
                          , loss='categorical_crossentropy'
                          , metrics=[metrics.CategoricalAccuracy()])
    
    
      history = autoencoder.fit([X_train]
       , [X_train]
       , epochs=16
       , batch_size=2
       , verbose=2
       , validation_data=((X_test),(X_test))
                         )
    

Thank you for any help!

$\endgroup$
2
  • $\begingroup$ OHE means?..... one hot?... $\endgroup$ Jun 26 at 12:49
  • $\begingroup$ Yes, using OHE as one hot encoding - thanks for reminding me to clarify $\endgroup$
    – user37649
    Jun 27 at 1:11

1 Answer 1

0
$\begingroup$

There is some confusion in your question, since you are talking about an AE, but you are using softmax as output... which makes almost no sense

First of all, the loss by itself, has no meaning:
If I give you two models with two different losses (in your case, two AE), you cannot, in any way, guess which is the best one (therefore a loss that is 20 means nothing)

About your questions:

  1. depends on the distribution... one hot encoding an output supposes a multinomial distribution, which you can approximate with both a softmax and sigmoid, but 99% of the times you should use a softmax, since encodes constraints of the distribution directly, and the NN don't have to learn them (like when approximating a probability, you can for sure use a linear output layer, but the NN has to learn to output values between 0 and 1, and there is no guarantee that it will always do that)
  2. an AE aims to reduce the dimensionality... you start with 4 features and you allow it to use 6 dimensions (in this case we are talking about overcomplete AE, but you need some constraint to make them work as expected)
  3. welcome in the world of non convex optimization... if the starting point is random (which is since the weights are initialized randomly), you will probably get to a different local minima
  4. an autoencoder aims to learn the identity function ($decoder(encoder(x)) = x$) therefore obviously no, the input has to have the same shape as the output

At this point, I'm pretty certain that an AE is not what you are looking for, but most likely a normal discriminative NN (since you are using cat-cross-ent as loss function)

$\endgroup$
3
  • $\begingroup$ I find the tone of this answer to be disappointing. On a certain level, you can compare losses between models - because if a loss is zero, then we say the model has predicted perfectly. 1. Why do you recommend softmax here, but dismiss it above for an AE? 2. The reduction in dimensionality is only in the middle layers, as the decoding reconstructs - I'm not sure how this answers the above 3. Thanks for clarifying this - you're right, I should have seeded the model 4. Ok point noted, so you're saying an AE has strict rules that surround it - but those are more human established. $\endgroup$
    – user37649
    Jun 27 at 1:13
  • $\begingroup$ @user37649 loss zero means that you model has just memorized the training set, and I'm saying that it's not clear what you aim to do... AE aims to do dimensionality reduction or density estimation, which has nothing to do with a softmax output layer, since that is used in discriminative model to specify the output category of a multinomial distribution $\endgroup$ Jun 27 at 10:38
  • $\begingroup$ Does the validation set give us an opportunity to measure the validation loss, such that we can infer the model performance? On softmax or any activation function, is that not a relationship of the input (and output) vectors? (ex: values are bound between [0,1], [-1,1], etc? $\endgroup$
    – user37649
    Jun 27 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.