What is the name of the following statistical / informational interaction:

given A, I know exactly what B is. given B, I know to some extent what A is.

I'm not looking for a probability but rather something like correlation. Something that tells me that I don't need to include B as a feature when A is already a feature. A correlation heatmap wouldn't give me this information, but there must be some computation that tells me "Don't include B, it's worthless."

To give some intuition: A could be an item_id and B an item_category.

Or am I wrong? Is it not worthless at all?


You might want to look at conditional entropy, H(A|B) and H(B|A).

  • $\begingroup$ Thank you. Will this be useful for feature selection? $\endgroup$ – Skusku Feb 12 '20 at 11:33
  • 1
    $\begingroup$ Assuming you can calculate those values (which means you have access to the marginals P(A), P(B), P(A,B) or a reasonable estimate thereof), yes. Given the two features A and B, if H(A|B) is low, you can use B as a feature and ignore A. Whether these computations are feasible is a separate question. $\endgroup$ – MrMulliner Feb 12 '20 at 11:41
  • 1
    $\begingroup$ Also, your ML model might benefit from having both, even if there's not technically any additional information there. $\endgroup$ – Ben Reiniger Feb 12 '20 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.