We usually filter out features (columns) that have low correlation or no significant impact on target variable. How would an algorithm, being trained with high dimensional data set (let’s say, more than thousands of features) contain rows with very high correlation but have different target variable, perform? Wouldn’t it make the ML algorithm confused in classification task?
Let me give a simple example to explain what I mean. Assume, we are given the price of a car and the task is to classify it as either of ‘Cheap Car’, ‘Budget Car’, ‘Luxury Car’, and ‘Elite Car’. Further assume, the distance between two rows is generally expected to be greater than 1000. For example, if a row describes a Car with price 1000, the next higher level car in our classification expected to be at least 2000. What if there is some anomaly in data set like a car with price 1000 classified as ‘Cheap’ whereas a car with price 1050 classified as ‘Elite’. That is grossly wrong. We eliminate irrelevant features. Shouldn’t there be something to eliminate confusing training examples?