How to know whether machine learning is possible for a given data set. I have been given a data set, I should check whether machine learning is possible or not for that data set. How can I do that. How do you come to conclusion that machine learning can be performed for the given data set or not.

  • $\begingroup$ Is the outcome from dataset known ? I.e. if Machine learning is applied, what should be the output. $\endgroup$ Mar 18 '19 at 6:58
  • $\begingroup$ Nope. We can use any of the features as the output . The requirement is to do any machine learning for the given data set. $\endgroup$ Mar 18 '19 at 8:58

For this feasibility study, following will be high level steps :

  1. For each feature perform PCA with rest of the features as train_x and feature as train_y. If you find a feature that can be predicted by other features; ML can be applied on Dataset
  2. Can a Human solve it ? As a person; can you find patterns for a given feature, based on other features ?
  3. Exploratory data analysis with Weka, Dataframe + Matplotlib or similar tools . https://datascienceguide.github.io/exploratory-data-analysis
  • $\begingroup$ Let me check that. $\endgroup$ Mar 18 '19 at 10:13
  • $\begingroup$ The first suggestion is not clear to me, can you open it? $\endgroup$
    – mcvkr
    Mar 18 '19 at 15:47
  • 1
    $\begingroup$ For example : Look at "A t-SNE plot of MNIST" exhibit in colah.github.io/posts/2014-10-Visualizing-MNIST . Since projection of dimensions seems to clearly sperates 9 classes (Digit feature in this context); it can be inferred that pixels can be used to predict digits. $\endgroup$ Mar 18 '19 at 17:53
  • $\begingroup$ Thanks for the example. $\endgroup$
    – mcvkr
    Mar 19 '19 at 1:43
  • $\begingroup$ For the first point, I checked whatever you said, but the accuracy is low for all the features . For the second point, no I am not able to get any insight into the data. So, probably we cannot do machine learning here. $\endgroup$ Mar 20 '19 at 4:36

I have asked the same question myself many times. To add a bit of a context: I work with relatively small data (~100 observations per experiment) of environmental nature, which is often sparse and/or imbalanced.

My empirical answer is very trivial: just try it! Also, do not forget that ML is a loosely defined term - some "classical" statistical tools may very well fall under it.

To begin, use the domain knowledge to set up appropriate research questions and think about what your data can tell you. Then start exploring the basics: draw a correlation plot, check whether the data is normally distributed, etc. Then you can apply unsupervised learning to look for more complicated relationships. Perhaps afterwards you may do some supervised one to make predictions.

One important remark. On one hand, do not get discouraged by poor PCA performance - it may pretty much happen that your data relationships are not linear. On the other hand, do not expect to build a Neural Network for every single problem ever - oftentimes they are not needed. Just go from a low level of complexity to a higher one, till it makes sense to continue.

Hope it helps!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.