0
$\begingroup$

I am trying to implement a paper in which the ultimate goal is to predict mutliple labels for instances (which are genes here). The feature matrix with shape of 1236*18930 is built by calculating term frequency of every gene's name in a set of scientific paper text and also is sparse. The dimentionality reduction is done by using a stacked denoising autoencoder (SDAE) before applying a feed forward neural network for multi label classification. The paper has only specified the number of output neurons for each layer of the SDAE which is 2700 and 400 and 60. And here is the implementation of the SDAE by me. In the encoded feature matrix which is the result of this AE, in average 20 columns are zero. So my question is, is my implementation reasonable according to the loss function? and the fact that some columns are 0, is a sign of fault in my implementation?

#adding noise
train_data_noised = np.zeros((1236, 18930))
threshold=0.2
a=int(18930*0.2)
print(a)
for i in range(feature_matrix.shape[0]):
  K = np.zeros(feature_matrix.shape[1])
  K[a:] = 1 #30 zeros
  np.random.shuffle(K)
  train_data_noised[i,:] = np.multiply(feature_matrix[i,:],K)
#train test split
X_train=feature_matrix[:1050]
X_test=feature_matrix[1050:]
train_noised=train_data_noised[:1050]
test_noised=train_data_noised[1050:]
ncol = train.shape[1]
encoding_dim = 60
input_dim = Input(shape = (ncol, ))

# Encoder Layers
encoded1 = Dense(2700, activation = 'relu')(input_dim)
encoded2 = Dense(400, activation = 'relu')(encoded1)
encoded13 = Dense(encoding_dim, activation = 'relu')(encoded2)

# Decoder Layers
decoded1 = Dense(400, activation = 'relu')(encoded13)
decoded2 = Dense(2700, activation = 'relu')(decoded1)
decoded13 = Dense(ncol, activation = 'sigmoid')(decoded2)

# Combine Encoder and Deocder layers
autoencoder = Model(inputs = input_dim, outputs = decoded13)

# Compile the Model
autoencoder.compile(    loss='binary_crossentropy',
    optimizer='adam')
autoencoder.summary()
autoencoder.fit(X_train, train_noised, epochs = 50, batch_size = 20, shuffle = True, validation_data = (X_test, test_noised))
#encoding feature matrix from 18930 to 60 dimentions
encoder = Model(inputs = input_dim, outputs = encoded13)
encoded_input = Input(shape = (encoding_dim, ))
encoded_feature_matrix=encoder.predict(feature_matrix)
```
$\endgroup$

1 Answer 1

1
$\begingroup$

The output layer of your encoder has a relu activation:

encoded13 = Dense(encoding_dim, activation = 'relu')(encoded2)

The relu is setting negative values in your latent vector to zero.

To fix this, remove the relu activation from your last encoder layer, and add the relu activation as the initial layer of your decoder.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.