0
$\begingroup$

I am using an autoencoder to detect anomalies in dataset of network traffic. The dataset is a csv file, and is loaded and preprocessed with pandas (encoded categorical features with pandas.get_dummies(...))

def categorical_normalization(dataframe, name):
  dummies = pd.get_dummies(dataframe[name])
  for x in dummies.columns:
    newname = f"{name}-{x}"
    dataframe[newname] = dummies[x]
  dataframe.drop(name, axis=1, inplace=True)

Therefore the process is taking too much ressources. After few research, I found tf.data. I've created a preprocessing layer to encode categorical features and normalize the dataset.

def preprocess_layer(dataset, feature_layer, normalizer_layer):
  dataset = dataset.map(lambda x, y: (feature_layer(x), y))
  norm = normalizer_layer(dataset)
  dataset = dataset.map(lambda x, y: (norm(x), y))
  return dataset.batch(batch_size)

I tired to pass x_train (tf.data.Dataset) in the autoencoder model.fit(x = x_train, y= x_val) its not working I also did dataset.map(lambda x, y: x, x) so it matches

Even inside the autoencoder model

# Autoencoder class      
def call(self, x):
    preprocessed_x = self.preprocess(x)
    encoded = self.encoder(preprocessed_x)
    decoded = self.decoder(encoded)
    return decoded

Got the same error ValueError: logits and labels must have the same shape ((None, 33) vs (None, 1))

And I receive this error with this code :

def model(preprocessing_head, inputs):
  body = AutoEncoder(33)

  preprocessed_inputs = preprocessing_head(inputs)
  result = body(preprocessed_inputs)
  model = tf.keras.Model(inputs, result)

  model.compile(loss=tf.losses.MeanSquaredError,
                optimizer=tf.optimizers.Adam())
  return model

anomaly = model(preprocessing_model, inputs)
anomaly.fit(x=features_dict, y=features_dict, epochs=10)

 ValueError: Found unexpected keys that do not correspond to any Model output: dict_keys([*features_keys...*]). Expected: ['auto_encoder_22']

How to preprocess the dataset and train it with the autoencoder

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.