I am using an autoencoder to detect anomalies in dataset of network traffic. The dataset is a csv file, and is loaded and preprocessed with pandas (encoded categorical features with pandas.get_dummies(...))

def categorical_normalization(dataframe, name):
  dummies = pd.get_dummies(dataframe[name])
  for x in dummies.columns:
    newname = f"{name}-{x}"
    dataframe[newname] = dummies[x]
  dataframe.drop(name, axis=1, inplace=True)

Therefore the process is taking too much ressources. After few research, I found tf.data. I've created a preprocessing layer to encode categorical features and normalize the dataset.

def preprocess_layer(dataset, feature_layer, normalizer_layer):
  dataset = dataset.map(lambda x, y: (feature_layer(x), y))
  norm = normalizer_layer(dataset)
  dataset = dataset.map(lambda x, y: (norm(x), y))
  return dataset.batch(batch_size)

I tired to pass x_train (tf.data.Dataset) in the autoencoder model.fit(x = x_train, y= x_val) its not working I also did dataset.map(lambda x, y: x, x) so it matches

Even inside the autoencoder model

# Autoencoder class      
def call(self, x):
    preprocessed_x = self.preprocess(x)
    encoded = self.encoder(preprocessed_x)
    decoded = self.decoder(encoded)
    return decoded

Got the same error ValueError: logits and labels must have the same shape ((None, 33) vs (None, 1))

And I receive this error with this code :

def model(preprocessing_head, inputs):
  body = AutoEncoder(33)

  preprocessed_inputs = preprocessing_head(inputs)
  result = body(preprocessed_inputs)
  model = tf.keras.Model(inputs, result)

  return model

anomaly = model(preprocessing_model, inputs)
anomaly.fit(x=features_dict, y=features_dict, epochs=10)

 ValueError: Found unexpected keys that do not correspond to any Model output: dict_keys([*features_keys...*]). Expected: ['auto_encoder_22']

How to preprocess the dataset and train it with the autoencoder


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.