# How to preprocess data?

What is the generic way to preprocess data for machine learning and predictive models ?What are the sequence of steps to be taken?

You are having a dataset with both continous and categorical data

1.Centre the data

for numerical variable centering usually done by subtracting mean of the column and some times by minimum value of the column

2.Scaling

Scaling the data means converting range of data between 0 and 1. It is done by different methods some follows by dividing with range and some follows by dividing with variance(unit var)

3.Skewness

Check for the skewness of the variable in given data.If the skewness factor is not around zero then try to perform data tranformations using exponential transformations , Box cox and logarithmic e.t.c

4.One on n encoding

Its is also called as one hot encoding also it is used to encode categorical variables that to nominal variables or else if you can use Equilateral encoding etc.

for ordinal variable you can encode them as increasing or decreasing order

5.Feature Selection and Importance

If you want to remove unwanted variables in your data then you use any feature selection algorithm like recursive feature selection or feature importance by trees etc.

6.Dimensionality Reduction

To reduce the number of dimensions in your data go with algorithms like Principal Component Analysis (unsupervised) or Partial Least Squares(Supervised) and select the number of dimensions which can nearly describe variance of your data.

7.Removal of outliers

Outliers are the portion of your data you are not explored in some situations to remove outliers you can go with techniques like Spatial Sign and some other techniques.

8.Missing Values

Missing values are the most common problem in data science.In order to over come to that you can impute values using different approaches like knn impute and build some model to predict missing data using other variables.

9.Binning Data

This is like converting continous data into categorical or interval data this sounds interesting but often leads to loss of valuable information

These are some of the important and basic steps of data preprocessing for majority of the algorithms but some algorithms doesn't need some of the steps like Random forest accepting factor values (so no need of one on n encoding) XgBoost accepting missing Values etc.

• I'm not sure if I like this: Posting your own question and answer. It could be your personal reference list, but then this is not the place. It could be a general reference list, but it has not sufficient quality. What is your intention? Commented Oct 28, 2017 at 13:58
• @Pieter21 I think you missed the feature of sharing things by Question and Answer style.Its for the starters to show how things has to be done If they a know a step name then it will be helpful for them.If a data science starter is from non mathematical background then there is no way getting these things . Commented Oct 29, 2017 at 13:33