11
$\begingroup$

I have many times analysed a dataset on which I could not really do any sort of classification. To see whether I can get a classifier I have usually used the following steps:

  1. Generate box plots of label against numerical values.
  2. Reduce the dimensionality to 2 or 3 to see if classes are separable, also tried LDA sometimes.
  3. Forcefully try to fit SVMs and Random Forests and look at feature-importance to see if the features make any sense or not.
  4. Try to change the balance of classes and techniques like under-sampling and over-sampling to check if class imbalance might be an issue.

There are many other approaches I can think of, but have not tried. Sometimes I know that these features are not good and not at all related to the label we are trying to predict. I then use that business intuition to end the exercise, concluding that we need better features or totally different labels.

My question is how does a Data Scientist report that the classification can not be done with these features. Is there any statistical way to report this or fitting the data in different algorithms first and looking at validation metric is the best option?

$\endgroup$
  • $\begingroup$ I feel like if this question has a clear and objective answer, then all of scientific endeavor becomes meaningless. Science is art. $\endgroup$ – Mephy Dec 5 '17 at 12:41
  • $\begingroup$ Haha yeah true. I am just interested to know more methods to find separability. More like how to statistically show to clients that putting more time on some exercises might be futile. $\endgroup$ – vc_dim Dec 5 '17 at 13:08
  • 1
    $\begingroup$ I suggest dropping the last two paragraphs, as they make the question too open/broad. The first part - how to handle and report a non-result professionally - should be answerable, and personally I think we need more of this type of Data Science professional etiquette question on the site. $\endgroup$ – Neil Slater Dec 5 '17 at 13:54
  • $\begingroup$ Makes sense Neil. Editing it to be crisp. $\endgroup$ – vc_dim Dec 5 '17 at 16:27
4
$\begingroup$

It depends on your data. There is something called human level error. Suppose tasks like reading of printed books, humans do not struggle to read and it might not happen to make a mistake unless because of bad quality of printing. In cases like reading hand-written manuscripts, it may happen a lot not to understand all words if the font of the writer is odd to reader. In the first situation the human level error is too low and the learning algorithms can have the same performance but the second example illustrates the fact that in some situations the human level error is so much high and in a usual manner (if you use the same features as humans) your learning algorithm will have so much error ratio.

In statistical learning, there is something called Bayes Error, whenever the distribution of classes overlap, the ratio of error is large. without changing the features, the Bayes error of the current distributions is the best performance and can not be reduced at all.

I also suggest you reading here. Problems with a large amount of Bayes error with appointed features are considered not classifiable with in the space of those features. As another example you can suppose you want to classify cars with lights on. If you try to do that in the morning, you yourself may have lots of errors and if you use same images for training the learning algorithm, that may have too.

Also I recommend you not to change the distribution of your classes. In such cases, the result of classifier near the boundary would be completely random. The distribution of data for training your machine learning algorithm should not be changed and should be as it is in the real condition.

| improve this answer | |
$\endgroup$
  • $\begingroup$ This is helpful, I did not know about Bayes Error. I did find difficulty in improving accuracies after changing the distribution of classes. I agree it is not a great idea. Still I sometimes have tried to change class hierarchies hoping that feature might represent more granular or abstract classes (EG. changing monkeys, dogs and other to Mammal). I think if there is a Bayes error, changing class hierarchy will not help too. $\endgroup$ – vc_dim Dec 5 '17 at 16:37
  • $\begingroup$ @SumitSinghChauhan Actually in situations where the Bayes error is large, trying to do feature engineering is the best solution. Because deep learning which finds the features itself, can not be used in cases where the dataset is not large. $\endgroup$ – Media Dec 5 '17 at 16:51
5
$\begingroup$

Take a sample element from one class and a sample element from the other class. Is it possible for these two elements to have the exact same feature vector? If that can ever happen then the two classes are not completely seperable using your current feature vectors (since the classification decision is based entirely in the feature vector for a given element).

On the other hand, if *every" element in one class has a corresponding element in the other class such that the two elements have the same feature vectors, then the two classes are indistinguishable using your current feature vectors.

Furthermore, if that condition holds for only some of your elements and not others, then you are somewhere in between, and you can use that as a basis to measure how well you can hope a classifier to perform using your current feature set.

All of these evaluations can be used to argue to varying degrees that you need to extract more features.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks Robert. This is familiar and seems right. I have done the same analysis for some of the projects. I had a dataset with same feature vector getting tagged differently and used that as a basis for how accurate I can be. Some times plotting box plot helped too. So far I have found the approach you just mentioned the best to convince clients. $\endgroup$ – vc_dim Dec 6 '17 at 8:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.