Feature has a pattern in relation to class but does not enhance classifier to predict class

Question

I have a feature x that when I plot it again my class variable y it shows some sort of pattern, i.e. it is obvious that x has a relation with y, but when I add the x to my logistic model it reduce the model accuracy. I want to know the reason, I thought that maybe there is a different distribution in my training and testing data, that lead to this, and checked this hypothesis, the result was that distributions are different. In general case any idea is appreciated.

Answer 1

There are multiple things at play here.

First of all, if your training and testing data are different then it becomes difficult to learn to generalize, because it will learn the relationships in your training data and if those relationships are significantly different then it will learn the wrong things. It could be that the data density is just different in your testing set which is not as much of a problem.

Second of all, if the relationship that you see is not linear, your logistic regression might not be able to pick up the relationship due to the underlying linear assumption.

Third of all, maybe other features are highly (co)correlated and adding the new feature does not add any signal but does add the potential for fitting additional noise, thus explaining the performance reduction.