0
$\begingroup$

What is the most common order of data cleaning, data transformation and exploratory data analysis?

For me it seems most logical to do data cleaning, then EDA and finally data transformation (encoding of categorical variables, and feature scaling).

Doing data transformation before the EDA, seems to make the EDA not that useful, as you cant ex. check for stuff like:

Passengers in the age interval 0-18 has a higher chance of survival

(if feature scaling has been applied to age feature).

But then again, doing data transformation after the EDA, also miss out on chance of encoding categorical variables and thereby visualize correlations of those with the target variable.

What is the order of the mentioned processes? And is there even an order?

$\endgroup$
2
$\begingroup$

Although not very helpful, the answer is probably "it depends".

I like to do data cleaning and some EDA together since EDA can highlight appropriate treatments to clean the data - e.g. influencing how to handle missing values.

I think data transformation should be done just prior to modelling; whether or not you need to do any transformation at all depends on the techniques you plan to use.

| improve this answer | |
$\endgroup$
0
$\begingroup$

I agree about "it depends" - upon your goal, and upon the nature of the data, and upon how much you and your team know about the data. For text data, the cleaning methods are pretty clear, so I would probably do data cleaning first. For high-volume image data (cancer screening, seismology), there are major trade-offs between data-reduction and feature-detection. In this high-volume domain, I think your pathway would be less clear - UNLESS you or your team know what methods of data-reduction and cleaning have been useful in the past. Domain knowledge can be very important in selecting and formatting the data for your subsequent analyses.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.