Hello I am using keras to develop a neural network model and I have a data of 45 numerical predictor variables, 2 categorical targets that will be predicted each with a different model. As I found, there is no feature importance model in keras. There are three options I can use, correlation ratio between the variables, kendals rank coefficient values and lasso regulation. Which one do you think is suitable to be used for feature selection in neural networks?
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
3
Unfortunately, there is no direct way to assess the "importance" of a variable in a Neural Network. One option, very time consuming, consists in removing each variable, one by one, replacing it with random noise, and checking how the performance changes. That will give you an idea on the contribution of a variable.
Alternatively, stick with importance scores of Tree-based models (such as Random Forests), or with good old statistical analysis. Shapley value regressions are a famous example.
-
$\begingroup$ I think I will be using correlation ratio and sort the variables accordingly. Any other suggestions? $\endgroup$– JeanCommented May 22, 2020 at 19:22
-
$\begingroup$ Why is random noise better than setting the feature (or associated weights) to 0? $\endgroup$– xi45Commented May 26, 2020 at 12:25
-
$\begingroup$ Substituting the actual variable with random noise is just a way to "shut down" that variable for that specific session. In that way you can counterfactually assess the importance of the missing variable. Noise by definition cannot be learned. I can't say if it's better or worse than your option, that is task-specific I guess. $\endgroup$– LeevoCommented May 26, 2020 at 13:53
$\begingroup$
$\endgroup$
0
For feature importance, you might want to consider using Shapley values or LIME. There are some examples here.
$\begingroup$
$\endgroup$
0
To complement the other two answers, you can use various model-agnostic methods to assess feature importance. See this nice e-book: Interpretable Machine Learning - A Guide for Making Black Box Models Explainable by Christoph Molna
$\begingroup$
$\endgroup$
As a solution, you can train single layer linear perceptron to find weights of each feature. After that iteratively drop the most useless feature (with min weight) and test combinations of remained features (with the highest accuracy) on your NN. Code below
def features_looker(X_tr, y_tr, X_ts, y_ts, remove_features=0):
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(units=1, input_dim=X_tr.shape[1], activation='sigmoid'))
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])
info = model.fit(X_tr, y_tr, epochs=30, batch_size=100, verbose=0, validation_data=(X_ts, y_ts))
accuracy = model.evaluate(X_tr, y_tr, verbose=0)[1]
val_accuracy = model.evaluate(X_ts, y_ts, verbose=0)[1]
col = X_tr.columns
weights = np.abs(model.layers[0].get_weights()[0].flatten())
weights_df = pd.DataFrame({'feature': col, 'weight': weights}).sort_values(by='weight', ascending=False)
worst_features = weights_df.nsmallest(remove_features, 'weight')['feature'].values
print('Features: ', len(col))
print('Train Accur: ', accuracy)
print('Test Accur: ', val_accuracy)
return info, worst_features, weights_df
