What is the name of the following statistical / informational interaction:

given A, I know exactly what B is. given B, I know to some extent what A is.

I'm not looking for a probability but rather something like correlation. Something that tells me that I don't need to include B as a feature when A is already a feature. A correlation heatmap wouldn't give me this information, but there must be some computation that tells me "Don't include B, it's worthless."

To give some intuition: A could be an item_id and B an item_category.

Or am I wrong? Is it not worthless at all?


1 Answer 1


You might want to look at conditional entropy, H(A|B) and H(B|A).

  • $\begingroup$ Thank you. Will this be useful for feature selection? $\endgroup$
    – Skusku
    Commented Feb 12, 2020 at 11:33
  • 1
    $\begingroup$ Assuming you can calculate those values (which means you have access to the marginals P(A), P(B), P(A,B) or a reasonable estimate thereof), yes. Given the two features A and B, if H(A|B) is low, you can use B as a feature and ignore A. Whether these computations are feasible is a separate question. $\endgroup$
    – MrMulliner
    Commented Feb 12, 2020 at 11:41
  • 1
    $\begingroup$ Also, your ML model might benefit from having both, even if there's not technically any additional information there. $\endgroup$
    – Ben Reiniger
    Commented Feb 12, 2020 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.