0
$\begingroup$

I'm trying to go through the first edition tabular challenge on Kaggle. Obviously my first few trials results did not satisfy me, so I went to see how other people did, and in the post of the first place winner I saw an interesting idea. He used a Deepstack Denoising Autoencoder to automatically feature engineer the dataset. The problem is that I understand the second part of the name (denoising autoencoder) but not the first. Or to be more specific: how he used it. To quote the part I have trouble understanding (from here):

The deepstack DAE transforms the input (500000 x 14) into an output of (500000 x 4500). If done correctly no more feature engineering is needed and stage two models should perform way better with this type of input data.

What I don't understand is that a line above that sentence he's saying that the

extraction of weights = new dataset

How? I mean, there are three hidden layers in his architecture:

dae = keras.models.Sequential([
    keras.layers.GaussianNoise(.1),
    keras.layers.Dense(14, activation='relu'),
    keras.layers.Dense(1500, activation='relu'),
    keras.layers.Dense(1500, activation='relu'),
    keras.layers.Dense(1500, activation='relu'),
    keras.layers.Dense(14, activation='relu'),
])

Hidden layer weights shapes are: 14x1500 + 1500 (for the bias), 1500x1500 + 1500, 1500x1500 + 1500 and however I add or multiply those numbers, they won't give me a dataset of 500 000 rows. So basically the question is - how do I do that?

EDIT

So I made a model and output extraction mechanisms, but I'm not sure if that's what I should be doing:

dae = keras.models.Sequential([
    keras.layers.GaussianNoise(.1),
    keras.layers.Dense(14, activation='relu'),
    keras.layers.Dense(1500, activation='relu', name='output_1'),
    keras.layers.Dense(1500, activation='relu', name='output_2'),
    keras.layers.Dense(1500, activation='relu', name='output_3'),
    keras.layers.Dense(14, activation='relu'),
])
early_stopping = keras.callbacks.EarlyStopping(monitor='val_loss', patience=10,
                                               min_delta=1e-4)

dae.compile(optimizer='adam', loss='mse')
dae.fit(X_train, X_train, epochs=200, batch_size=64,
        callbacks=[early_stopping],
        validation_data=(X_valid, X_valid))

And I'm using its hidden layers like so:

output_1 = K.function([dae.get_layer('output_1').input],
                      [dae.get_layer('output_1').output])
output_2 = K.function([dae.get_layer('output_2').input],
                      [dae.get_layer('output_2').output])
output_3 = K.function([dae.get_layer('output_3').input],
                      [dae.get_layer('output_3').output])
new_train_list = []

for row in train:
    row_transposed = row.reshape(1, 15)
    features, target = row_transposed[0, :-1].reshape(1, 14), row_transposed[0, -1]
    new_features_1 = output_1(features)
    new_features_2 = output_2(new_features_1)
    new_features_3 = output_3(new_features_2)

    new_train_list.append(
        np.hstack((
            new_features_1[0],
            new_features_2[0],
            new_features_3[0],
            np.array(target).reshape(1, 1))))

    if len(new_train_list) % 1000 == 0:
        print(f'Processed {len(new_train_list)} of {train.shape[0]} total rows.')

new_train_ds = np.array(new_train_list)
$\endgroup$

1 Answer 1

1
$\begingroup$

In fact, what was done was to take the output of each of the three hidden layers in the DAE and concatenate them together to form the new feature vector. The size of the output of each hidden layer is 1500, and so, the size of the new feature vector is 3x1500 = 4500.

The line

extraction of weights = new dataset

is misleading.

$\endgroup$
4
  • $\begingroup$ So in order to achieve what he achieved I need to run my 500 000 rows three times - one for each of the layers? $\endgroup$
    – Marek M.
    Apr 4, 2023 at 18:09
  • $\begingroup$ You need to evaluate your autoencoder one time for each row. Each row passes through the three hidden layers of the autoencoder, each hidden layer outputs a vector of size 1500. Thus, 3x1500 = 4500 features per row. Your new dataset is of size 500000x4500. $\endgroup$
    – Asterion
    Apr 4, 2023 at 19:51
  • $\begingroup$ Ok, I’ll try that in the morning and if it works I’ll accept your answer. $\endgroup$
    – Marek M.
    Apr 4, 2023 at 20:28
  • $\begingroup$ So I updated the question, bc I'm still not sure if that's what's it all about. Could you please take a look at the updated question? $\endgroup$
    – Marek M.
    Apr 5, 2023 at 7:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.