I have a binary classification problem where most of the features are categorical with 4 possible values: yes, no, irrelevant, nan. I am trying to find the modular global feature importance of those features to the target column (which is binary).

The columns represent checkups in a certain procedure. If there was any problem in a certain feature the value would be "yes", and otherwise it would be "no". If for a certain procedure, a certain checkup is irrelevant, the answer will be "irrelevant".

| Feature 1 | Feature 2  | Target |

| --------- | ---------- | ------ |

| Yes       | Irrelevant | Yes    |

| No        | Yes        | Yes    |

| NaN       | No         | No     |

How should I treat the irrelevant values?

  • $\begingroup$ first of all, try to format your question as supposed to... second, modular global feature importance? what you mean? $\endgroup$
    – anon
    Jun 26, 2022 at 13:04
  • $\begingroup$ I guess it is a long version for "global importance", as in doi.org/10.1007/s42452-021-04148-9: "While a modular global feature importance measures the importance of the feature for the entire model, a local importance measures the contribution of the feature for a specific observation. " $\endgroup$
    – Ohm
    Jun 27, 2022 at 14:15
  • 2
    $\begingroup$ One can only begin answering this question after they understand what "irrelevant" might mean in the context of the dataset. Does it mean that for certain observations, that feature is irrelevant to the prediction of the target variable? You might want to give us some more information about the dataset. $\endgroup$ Jun 28, 2022 at 11:38

2 Answers 2


The following answer is based on the reasonable assumption that for a certain (medical?) procedure, an irrelevant checkup(feature) always remains irrelevant. Eg: The procedure of checking for a fracture, a certain kidney checkup always remains irrelevant. I also assume that each procedure has a good amount of representation in the dataset.

Assuming Procedure type is known:

If you already have a column that tells about what procedure was being performed, you'd know for sure that all the instances of this procedure will have a specific set of columns as irrelevant. You can therefore perform a .groupby() to group these procedures. For a given procedure, You'd have some features that are always 'irrelevant' and other features which are never 'irrelevant'. Now, for every grouped dataset, you can perform Permutation Feature Importance from sklearn or ELI5 to get the feature importance of all features and can simply discard the features that were always 'irrelevant' (values for them will be 0 anyways, since permutating them doesn't change anything.).

You can aggregate different feature importances of the same feature either using the arithmetic mean or formulate it to also account for the number of occurrences of a feature across different procedures(for eg: if feature A occurs seven times(ie in seven procedures) with medium FI and feature B occurs only once(ie in one procedure) with high FI, you might want to give more weightage to A than B when calculating FI for the entire dataset).

Assuming Procedure type is unknown:

Even if this is the case, if the first assumption is true, you can still bucket instances based on the unique features that are irrelevant, consider the following example:

Instance # feature A feature B feature C
1 Yes No Irrelevant
2 No Irrelevant Irrelevant
3 Yes Yes Irrelevant
4 Yes Irrelevant Irrelevant
5 No No No

Then you can approximate different procedures based on unique patterns of features with 'irrelevant' values. In the above example that would give the grouping as $[\{1,3\}, \{2, 4\}, \{5\}]$. Then you can proceed in the same way as described in the first part.


Permutation feature importance is a way to access global feature importance. Permutation feature importance is defined as the decrease in an evaluation metric when a feature value is randomly shuffled, breaking the relationship between the feature and the target. Any reduction in an evaluation metric is indicative of how much the model depends on the feature. Permutation feature importance can be used with any machine learning algorithm (assuming tabular data).

One option would be to feed all the information into the permutation feature importance algorithm, letting the algorithm figure out the empirical importance of each feature for this task.

Even though a feature is labeled as "Irrelevant", it might still have some importance for prediction.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.