I am new to machine learning and have finished Andrew Ng's course on Coursera. I have just begun to tackle my first "real" ML problem - which is a binary classification problem. I was wondering if the steps I have taken is the "usual" way to work through a problem.

  1. I have gathered my data which is in this case is some images (positive and negative images)

  2. Preprocess the data - in my case read the images and convert these to grayscale images in order to simplify the data. I have then set up my arrays in terms of input and labels. No feature scaling is needed in my opinion or anyother preprocessing.

After this I split the data in training and test set (80-20 ratio). I used stratified split.

  1. Model selection and evaluation. In this case I used three models: logistic reg., SGDClassifier and a decision tree.

  2. Train the different models and evaluate them using k-fold cross validation sets. I evaluated the mean accuracy on the CV set and also the F1-score on the CV set. Based on this I selected the best modell (the scores for all the modells on the training set was 1 and on the CV sets 0.91 (log.reg) , 0,57 (dec.tree) and 0.73 (SGD))

NOTE : Shouldn't I plot some learning curves too see if there is a overfitt or underfitt problem (based on the result from the CV sets it seems like log.reg model is a good fit). Is there anything else to study or is it enough based on the scores?

  1. Fine tune the selected model.

  2. Evaluate on the test set.

  1. Depending on the images you used but I would not necessarily convert them to greyscale as there could be vital information in the colors of the image when it comes to making a prediction.

  2. Feature scaling is not necessary for any of those algorithms but they could perform better with it.

  3. Yes you should plot the Receiver Operating Characteristic and the Area Under the Curve. This is a metric you should be using as your problem is about Binary Classification.

The rest looks good.

  • $\begingroup$ 1. How do you decide if there is some vital information. I guess running a model without the grayscale transformation is one solution. But again in this case the contrast in the grayscale images should be enough for the model to distinquish between false and positve. 3. Regarding the ROC and AUC why is it neccesary to plot these? Wouldnt the accuracy or the F1 score be enough to decide which model generalizes best? $\endgroup$
    – John
    Sep 12 '20 at 16:56
  • $\begingroup$ That's true its not necessary to keep them as RGB images. $\endgroup$
    – yudhiesh
    Sep 12 '20 at 17:02
  • $\begingroup$ For me when I train binary classification models I tend to plot F1 score, Precision, Recall and ROC & AUC. F1 score is better suited when you have a class imbalance. But if both classes make up 50% of your dataset, or both make up a sizable fraction, and you care about your performance in identifying each class equally, then you should use the AUC, which optimizes for both classes, positive and negative. $\endgroup$
    – yudhiesh
    Sep 12 '20 at 17:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.