0
$\begingroup$

I was wondering if I have the correct order of preprocessing/EDA/feature engineering below?

Yes there are nuances and may vary from problem to problem, but am just looking for a general pipeline for 90% of machine learning problems I will encounter:

enter image description here

Full resolution here: https://lucid.app/publicSegments/view/6ef67134-eaa4-4043-8a26-158f4ff8e0e4/image.png

$\endgroup$

1 Answer 1

2
$\begingroup$

I think you will find an answer to your questions in this paper:

Biswas, S., Wardat, M., & Rajan, H. (2022, May). The art and practice of data science pipelines: A comprehensive study of data science pipelines in theory, in-the-small, and in-the-large. In Proceedings of the 44th International Conference on Software Engineering (pp. 2091-2103).

The authors analysed many different data science pipelines (among others 21 best GitHub projects in this area). They extracted and coined a few proposals for standards, I'm sure you will find something for yourself. For instance - you can read what are the distributions of order of stages between each other, how often some stages are used in general or what is the most popular pipeline.

$\endgroup$
1
  • $\begingroup$ Thank you, just read the paper. However, it does not seem like it dives deeply into the Data Preparation stage, which is what I am looking for (proper order of operations in EDA/Data Preparation/Feature Engineering stage) , especially in the Large Models section of the paper (which is most representative of the real world practical data science pipeline). $\endgroup$
    – Katsu
    Nov 28, 2023 at 22:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.