10
$\begingroup$

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own process by looking at a lot of tutorials that all do it a slightly different way.

I would like to have a standard process to go by, and once I am comfortable with it, I can choose to deviate. I'd like some input from you pillars of the industry. Is this a good routine for a beginner to follow?

  1. Get Data
  2. Clean Data
  3. Split data into Training and Test Data ~(80/20)
  4. Separately, for training and test sets:
    1. Normalize Data (continuous features):
      • standardize (divide by std. deviation)
      • center (subtract mean)
    2. Impute missing values
    3. Feature Engineering
    4. Encode Categorical Variables:
      • Integer Encoding
      • One Hot Encoding
      • Target Encoding
      • Weight of Evidence
  5. Separate labels from Test set if classification problem. Keep aside.
  6. Choose a few models.
  7. for each model, using k-fold cross-validaton:
    1. Train base model on "training set".
    2. Tune and test hyper parameters on "validation set"
    3. Save best scores and parameters
  8. Compare each model's final scores on the never touched test data

  9. Choose the model with highest scores.

Edit: Thank you for the overwhelming number of responses. Lots of times my questions get a single answer or none at all. I appreciate the time taken to help out a beginner.

I have edited the steps above to reflect the wonderful answers below. I hope that this helps another beginner somewhere else.

$\endgroup$
  • 1
    $\begingroup$ what is integer encoding_ $\endgroup$ – Carlos Mougan Apr 6 at 9:46
  • 1
    $\begingroup$ @CarlosMougan I he refers to what sklearn calls label encoding: scikit-learn.org/stable/modules/generated/… $\endgroup$ – Simon Larsson Apr 6 at 9:55
  • 1
    $\begingroup$ I suggest you think more about outlining substeps on step 3. It will be decisive to your model performance, sometimes even more than efforts on modelling. Also, it's important to get your step 6 before it to avoid "data leakage" as @Simon Larsson mentioned in his answer. $\endgroup$ – Adelson Araújo Apr 6 at 16:22
  • 1
    $\begingroup$ When you correct what you wrote in your question in response to answers, you invalidate the answers and kind of undermine the whole Q&A format. It would be better to just leave it as is or post an answer with the fixed process (assuming this adds something to the answers). Although more specific questions like "when should I split data into training and test sets" would be a better fit for the Q&A format than "please review this". The former could result in a few more questions, but this is not a problem at all. The latter would work better in a discussion forum. $\endgroup$ – NotThatGuy Apr 7 at 7:21
  • 1
    $\begingroup$ Perhaps a side note on platform choice, and which one to use might help. i.e. Tensorflow, Caffe, etc. And some consideration may be required for deployment to mobile (which in Tensorflow requires some conversion/additional support). $\endgroup$ – Emile Apr 9 at 10:01
13
$\begingroup$

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the split at 3. in your flow chart.

A common step I think you have missed is imputation of missing values. I would put that before feature engineering.

Overall I think this is a good rough outline for a beginner to follow. It is overly simplistic and leaves a lot out, but I think you know that and you have to start somewhere.

| improve this answer | |
$\endgroup$
  • $\begingroup$ @SimonLarson Thank you! Great answer! 1. I totally see how normalizing the combined dataset is bad now! I will normalize train/validate/test separately from now on. 2. I kind of thought imputation was part of the data cleaning step, so that's why I left it out. I have been struggling with that part specifically. I think I'm being clever with imputations and feature engineering, but my model gets stays the same or gets worse 9/10 times. Same goes for hyper-parameter tuning. Perhaps my data-leaks are the issue? Thank you again for your answer! $\endgroup$ – rocksNwaves Apr 6 at 18:23
  • $\begingroup$ One followup question: If I split the data at step 3, that means I should impute missing values for each split separately. That makes sense if I'm using average values to fill in the blanks, so data-leaks don't occur. For feature engineering, There doesn't seem to be as much of a chance for data-leaks, so maybe I could make that step 3, before the split? Or am I wrong? $\endgroup$ – rocksNwaves Apr 6 at 18:37
  • 2
    $\begingroup$ Any answer on your first question would just be speculation, but I don't think it is due to a data leak. A data leak will not make your model worse until it gets used on truly unseen data. For your second question, it depends on what feature engineering you do. If you create a feature based on the mean of another feature you will get a leak. But splitting earlier makes it so you don't have think about this. $\endgroup$ – Simon Larsson Apr 6 at 19:15
  • $\begingroup$ @rocksNwaves Note you can't do normalization or missing data imputation on the test set; when you will apply your algorithm, there is no test set to compute averages with which to normalize, you have a single example in input and you have to apply it. What you do is compute the necessary statistics in the train set (e.g. for normalization, typically mean and standard deviation) and apply the saved numbers for test set normalization (so you cannot recompute mean and std.dev on the test set, but have to use the values coming from the train set) $\endgroup$ – Ant Apr 7 at 17:13
3
$\begingroup$

Yes, these are the basics step. Then in each step there is a lot more. If you want to get a bit deeper you can follow this book of Andriy Burkov of Machine Learning engineering

A couple notes in your process:

Before get data I Will put, define the question to resolve or something similar, but maybe this parted is granted.

Feature Engineering is one of the most important thing in ML, so probably spending a bit more of time there would help.

Normalize data helps mainly in Linear models, decision trees model has little/no effect.

Integer/Label Encoding is not specially good, there are better things as Target Encoding and Weight of Evidence encoding, have a look.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you, I'll take a look at Target/Weight encoding. I will also add the part about defining the question, because I think that is important. I've been working on Kaggle data-sets, where the question is pre-defined, but I see how that's not always the case. $\endgroup$ – rocksNwaves Apr 6 at 18:26
3
$\begingroup$

After 12 "Choose the model with highest scores." Maybe add "create ensemble of models" and try to improve accuracy further.

| improve this answer | |
$\endgroup$
2
$\begingroup$

Is this a good routine for a beginner to follow?

Yes, it's very good.

You could add:

  • K-fold Cross-validation("Split Training into Training and Test Data")
  • Feature selection before "Choose a few models."
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you @fuwiak, I have been practicing cross-validation with a single split to my training data. Yesterday I started learning and applying k-fold cross validation. I will add those steps :) $\endgroup$ – rocksNwaves Apr 6 at 18:24
  • 1
    $\begingroup$ My bad, I mean k-fold, I've just edited. $\endgroup$ – fuwiak Apr 6 at 18:26
1
$\begingroup$

Is this the end-to-end process?

  • Most importantly, you also need to understand the data you are using. It's not supposed to be a meat-grinder. Add some uni and multivariate analysis just before splitting your data. Look at the distributions and frequencies.
  • After you split 70/30 or 80/20 or whatever, are the distributions approximately similar?
  • I think you should also add touching base with stakeholders/business people just after feature engineering (and maybe add a loop arrow to reflect their feedback).
  • Another user mentioned ensemble models / model averaging at the end - I think that is also important. Wouldn't an ensemble model perform better that any single model?
  • You are also missing documentation - where are you documenting your steps? Is it all in your mind? How will others follow what you are doing?
  • What about four-eyes check aka pair programming?
  • What about version control? In most industries you will need to show how your models were derived and how they perform against alternatives.
  • What about edge cases for reasonable results for the best 2-3 models
  • Model explainability - how can you or your users trust the model without understanding how it is operating.
| improve this answer | |
$\endgroup$
  • $\begingroup$ Hey, these are some good suggestions! Some of them seem like they are for those who have already obtained a job in the field, and some of them apply to people like me who have just been learning. I'll keep it all in mind! $\endgroup$ – rocksNwaves Apr 11 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.