Unused bottleneck neurons in autoencoder

I'm using an autoencoder to compress categorical data, by a factor of about 20x.

For this part of my data set, I have roughly 3500 variables, so my final bottleneck size is 180. I'm getting pretty good losses & accuracy, but that's not what I'm asking about.

After looking at my compressed data, it turns out that quite a few of the encoded variables are zeros. In fact, out of 180 encoded variables, 111 of them are all zeros.

I reran my code twice to confirm that it wasn't a fluke (same parameters, same data, etc), and got similar results: 110 and 96 variables all zeros.

Does this mean I could reduce the bottleneck size by about 100 and still see basically the same losses and accuracy? Are these neurons just unused by my autoencoder?

Here is the code I use to generate it:

x_arr = np.array(x)
x_tr = np.transpose(x_arr)
test_index = np.random.choice(range(len(x_tr)), math.floor(len(x_tr)/5), replace=False)
x_test = to_categorical(x_tr[test_index])
x_train = to_categorical(np.delete(x_tr, test_index, 0))
print("Total data is of shape "+str(x_tr.shape))
print("Training data is of shape "+str(x_train.shape))
print("Testing data is of shape " + str(x_test.shape))
Total data is of shape (2548, 3593)
Training data is of shape (2039, 3593, 4)
Testing data is of shape (509, 3593, 4)

input_data = Input(shape=x_train[1].shape)
f = Flatten()(input_data)
hidden_1 = Dense(hidden_1_size, activation='relu')(f)
hidden_2 = Dense(hidden_2_size, activation='relu')(hidden_1)

code = Dense(code_size, activation='relu')(hidden_2)

hidden_2_rev = Dense(hidden_2_size, activation='relu')(code)
hidden_1_rev = Dense(hidden_1_size, activation='relu')(hidden_2_rev)
output_data = Dense(input_size*4, activation='softmax')(hidden_1_rev)
rshp_output_data = Reshape((-1,4))(output_data)

autoencoder = Model(input_data, rshp_output_data)
encoder = Model(input_data, code)

autoencoder.compile(optimizer='adam', loss='categorical_crossentropy', metrics=[tf.keras.metrics.Accuracy()]) # when data is OHE

history = autoencoder.fit( x_train, x_train, epochs=1000, shuffle=True, validation_data=(x_test, x_test),
callbacks=kcb.EarlyStopping(monitor="val_loss", patience=5, restore_best_weights=True) )

np.savetxt(outdir + "encoded_"+chromosome+"_"+str(start)+"_"+str(end)+".csv", np.transpose(np.array(encoder(to_categorical(x_tr)))), header=';'.join(sampleIDs), delimiter=';')

• This may well be an implementation error and your autoencoder may be performing well despite it. Maybe if you provide more information and we can see the code, we could provide a better assessment. – noe May 5 at 14:21