Force network to weigh specific variables during learning

Question

I have a pandas data frame containing around 100000 observations of plant species and their age with additional numerical predictors (climate). I used tensorflow and keras to build a sequential model to predict species age. Here is the code for my basic regression network:

# Create model structure
model = Sequential([
    Dense(128,activation='relu',
          kernel_regularizer=regularizers.L1L2(l1=1e-3,l2=1e-4)),
    Dense(64,activation='relu',
          kernel_regularizer=regularizers.L1L2(l1=1e-3,l2=1e-4)),
    Dense(32,activation='relu',
          kernel_regularizer=regularizers.L1L2(l1=1e-3,l2=1e-4)),
    Dense(16,activation='relu',
          kernel_regularizer=regularizers.L1L2(l1=1e-3,l2=1e-4)),
    Dense(1,activation='relu')
])

# Compile the models
model.compile(optimizer=tf.keras.optimizers.Adam(),
                     loss=tf.keras.losses.mae,
                     metrics=['mae'])

# Train
random.seed(132)
model_trained = model.fit(X_train,
                          y_train,
                          epochs = 200,
                          validation_split = 0.15,
                          verbose = 0)

I one hot encode the species data and normalize all variables before training. There is around 110 species, so after one hot encoding them, I have a training dataset with around 900000 rows and 180 columns.

For training, I thought that including data for all the species in one data frame, will allow better performance, because the model is learning from more data at once.

The model does not show signs of overfitting and the validation, testing and training MAE is all very similar. Seems good!

However, when I try and make predictions, using different numerical data (future climate), the age predictions are all the same for different species. E.g. When I set up my new data frame for prediction, I encode the data frame to be for a specific species i.e. Species_1 is present, therefore all other species are not.

# Species_1 prediction
[[23.043112]
 [23.11334 ]
 [24.231022]
 [23.026756]
 [25.771097]]

# Species_2 prediction
[[23.043112]
 [23.11334 ]
 [24.231022]
 [23.026756]
 [25.771097]]

The fact that the predictions for different species are identical, makes me think that the network did not use the one hot encoded species information during training, or rather that the species information was not considered important in predicting age and therefore received a lower importance. I am by no means an expert in neural networks, and I am still learning.

My goal is to make age predictions for different species under different climate scenarios. How do I force the network to always use the species data during training? Am I wrong to use all species data in one data frame and should rather split the data frames per species separately? Should I rather use other machine learning algorithms (Random Forest)? Is the network too complex, and therefore memorizes the response variable, or the relationship between influential predictors and the response? Any tips would be appreciated.

Answer 1

After gaining better understanding of the problem at hand, here is my solution to deep encode my species categories so that the network can learn from them.

# Perform deep encoding on species names

# Create test data frame containing y and Species
test = pd.concat([y, X['Species']], axis=1)

# Convert species names to ordinal numbers
test['Species_ord'] = pd.Categorical(test['Species'], categories=test['Species'].unique(), ordered=True).codes

# Define embedding parameters
m = len(test['Species'].unique())
embedding_size = min(50,m+1/2)

# Create model structure
model = Sequential([
    Embedding(input_dim = m,
              output_dim = embedding_size,
              input_length = 1,
              name="embedding"),
    Flatten(),
    Dense(32,activation="relu"),
    Dense(1,activation="relu")
])

# Compile the models
model.compile(optimizer=tf.keras.optimizers.Adam(),
              loss=tf.keras.losses.mae,
              metrics=['mae'])

# Train
random.seed(132)
model.fit(x = test['Species_ord'],
          y = test['Age'],
          epochs = 50,
          batch_size = 16,
          verbose = 1)

# Grab the new embedded variables for species
Species_embedded = model.get_layer('embedding').get_weights()
Species_embedded_df = pd.DataFrame(Species_embedded[0])

# Change name of embeddings
Species_embedded_df = Species_embedded_df.rename(columns=lambda x: f"Species_encode_{x}")

# Add species names to embeddings as ID
Species_embedded_df["Species"] = test['Species'].unique()

# Save to disk for later use
Species_embedded_df.to_csv(dir + "3-Resultats/Tables/Tree_Species_Embeddings.csv", index=False)

```