1
$\begingroup$

We usually filter out features (columns) that have low correlation or no significant impact on target variable. How would an algorithm, being trained with high dimensional data set (let’s say, more than thousands of features) contain rows with very high correlation but have different target variable, perform? Wouldn’t it make the ML algorithm confused in classification task?

Let me give a simple example to explain what I mean. Assume, we are given the price of a car and the task is to classify it as either of ‘Cheap Car’, ‘Budget Car’, ‘Luxury Car’, and ‘Elite Car’. Further assume, the distance between two rows is generally expected to be greater than 1000. For example, if a row describes a Car with price 1000, the next higher level car in our classification expected to be at least 2000. What if there is some anomaly in data set like a car with price 1000 classified as ‘Cheap’ whereas a car with price 1050 classified as ‘Elite’. That is grossly wrong. We eliminate irrelevant features. Shouldn’t there be something to eliminate confusing training examples?

$\endgroup$
8
  • $\begingroup$ You want to be careful. In your question you say that the car is misclassified as though it is wrong. There is a HUGE difference between a misclassified instance (human error) and the real distribution of two classes which overlap each other. Which one are you talking about? $\endgroup$
    – JahKnows
    Oct 27 '18 at 2:07
  • $\begingroup$ I am not sure I understand you. In my example, I indicated to the unexpectedly introduced anomaly in training examples which might be due to human error leading to misclassification or other consequences in the classification task. $\endgroup$ Oct 27 '18 at 2:18
  • $\begingroup$ As a simpler question. Are any of the recorded values in your dataset wrong? For example did an employee making observations about a car get tired and make mistakes writting numbers or classifying instances? OR are all the values correct and the different classes just happen to have some similar instances due to their nature. $\endgroup$
    – JahKnows
    Oct 27 '18 at 2:22
  • $\begingroup$ Oh, I get it now. Yes, different classes just happen to have some similar instances due to their nature. Actually, I am working on a customized domain-specific text classification problem. This classification is based on some highly changing variable i.e. human activity. A certain body of text might be classified as, say, 'CLASS1' at a time. The target is, when similar text body is inputted, it should present the class. But the problem is, a very similar body of text can be classified as 'CLASS2' in another time in future which doesn't invalidate earlier classification. $\endgroup$ Oct 27 '18 at 2:40
  • 1
    $\begingroup$ If the instances are not errors, then you SHOULD NOT discrad them. They are a part of the real distribution of your inputs. Thus they need to be included. If you discard them this will introduce bias to your model. THIS IS BAD. Try adding additional features, or perform a feature transformation to get a new feature space where the classes might be more seperable. $\endgroup$
    – JahKnows
    Oct 27 '18 at 3:14
1
$\begingroup$

The answer is yes, highly similar instances in your dataset that have different target classes will cause your model to perform poorly.

The reason for this is at the core of how all classification machine learning algorithms work. The goal of a classifier is find a function which can separate the two classes. Thus, if these two classes are very mixed then the probability of making a classification error increases and thus you will lose precision in your resulting classification.

One method to correct this problem is to add more features to your dataset. You should try to find features which will distance the distributions of these two classes. For example if classifying cats and dogs, it would not be a good idea to use features such as: number of legs, number of eyes, etc. This will cause the classes to be indistinguishable. Try adding features such as: weight, frequency of cry, etc. This can be difficult to do often as collecting additional data is expensive. You can also try to transform your data to a new feature space. A transformation mapping your features to a different space can cause their distributions to distance themselves.

$\endgroup$
2
  • $\begingroup$ Thanks. That's what I was thinking. Is there any recommended way to remove such highly correlated training examples like there are for feature elimination? $\endgroup$ Oct 27 '18 at 1:54
  • $\begingroup$ I'll edit the answer. $\endgroup$
    – JahKnows
    Oct 27 '18 at 1:57
0
$\begingroup$

What you describe is behaviour at the boundary of two classes. There would be legitimate cases where two data points with similar features might have different class labels. The change from one class label to another has to happen somewhere along the feature, if it has anything to do with the class label. Eg: At some point, the car would stop being a 'Budget' car, and start being a 'Luxury' car. Depending on how the dataset was constructed, i.e., labeled by the same person at differnt points in time, or labeled by different people, etc., and if the domain has any data points near the boundary, you can expect the feature intervals of different classes to overlap to some extent.

Robust models generally can handle these boundary conditions, and should be "fault-tolerant" so that they don't have unstable or unexpected behaviour due to these cases. For example in SVMs, regularization allows some classification errors to be made, so that a more robust boundary can be drawn between two classes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.