1
$\begingroup$

I'm having a very hard categorizing different methods of data preparation into the 3 categories Preprocessing, Data Engineering, Feature Engineering.

A somewhat common definition describes Feature Engineering as the creation of new feature from raw data using domain knowledge. (a )

Preprocessing is commonly described as "cleaning the data". (a)

But sometimes Feature Engineering is also described as including basic cleaning techniques like handling missing values (b).

In this post by google Preprocessing is divided into Data Engineering and Feature Engineering. Data Engineering here is really just putting the data in a form that is recognizable by some model and Feature Engineering is everything else (i.e feature scaling)

$\endgroup$

2 Answers 2

1
$\begingroup$

I think about the distinction in these terms. Suppose we already have the data stored in a database and want to train a good ML model on it. This involves three steps:

  1. move data to the right computers where the model will be trained (Data Engineering)

  2. format data in the right way such that model could be trained on it (Data Preprocessing)

  3. try various transformations of data values that allow us to train the best model, without altering the model's hyperparameters or training algorithm (Feature Engineering).

Step 1 can get complex when the dataset is massive or streaming in real-time, and involves lots of infrastructure work and data pipelining. Step 2 (Data Preprocessing) refers to basic steps that need to be taken to even get the data in a format that is compatible input to our ML model (eg. vector for a feedforward neural net). In Step 3 (Feature Engineering), one applies transformations to the data values (rescaling numerical values, creating interaction terms, encoding categorical/text features, etc), trying to find the best new representation of the data that when input to our model allows the model to be easily trained to high accuracy.

Some call this process ELT (Extract, Load, Transform) or ETL. But in this space it is usually assumed the optimal feature transformations to get best model are already known, and the goal is to execute Steps 1-3 as efficiently as possible within a single pipeline to quickly go from raw data to well-trained model.

$\endgroup$
0
$\begingroup$

I asked your question to the Data Engineer Consultant GPT to get this answer:

Your challenge in categorizing methods of data preparation into Preprocessing, Data Engineering, and Feature Engineering is quite common, as the boundaries between these areas can sometimes be blurred. Let's clarify each of these:

  1. Preprocessing: This is a broad category that includes all the initial transformations and cleaning operations on raw data to make it suitable for further analysis or modeling. It's about getting your data to a state where it can be used effectively. This can include tasks like handling missing values, normalizing data, encoding categorical variables, and removing duplicates. Preprocessing ensures that the data is in a clean, standardized format.

  2. Data Engineering: This area focuses on the technical and architectural aspects of handling data. It involves developing and managing the data pipeline, which includes data collection, storage, and retrieval. In the context of preprocessing as mentioned in the Google post, Data Engineering is about structuring and transforming raw data into a format that is usable by models. This might involve data integration, building data warehouses, and ensuring that data flows smoothly from source to destination.

  3. Feature Engineering: This is the process of using domain knowledge to extract new features from raw data. These features are designed to improve the performance of machine learning models. It's more than just cleaning the data; it's about creating new data points that didn't exist before, which might offer better insights or predictive power. This can include aggregating data, creating interaction terms, or applying domain-specific transformations.

The confusion often arises because there is some overlap. For example, handling missing values can be seen as both preprocessing and feature engineering. In a strict sense, replacing missing values might be preprocessing, but using the pattern of missing values to create a new feature (like a binary indicator for whether a value was missing) would fall under feature engineering.

The key is to understand that these categories are not rigid, and some tasks might fit into more than one. The most important aspect is to ensure that the data is in the best possible shape for analysis or modeling, regardless of the specific label you put on the process.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.