1
$\begingroup$

Consider the code below:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
import pandas as pd

data = pd.read_csv("data.csv")
x = data.drop("Price", axis = 1)
y = data.Price
xTrain, xTest, yTrain, yTest = train_test_split(x, y, test_size = 0.2, random_state = 1)

It simply reads a csv file and splits it into inputs and outputs, for training and testing, which can now be used to train a price prediction model using some algorithm like Random Forest or KNN or Linear Regression or something else. But let's say I have a feeling that scaling the data (like for example using MinMaxScaler) will yield better results. So do I scale the entire data. Or do I scale xTrain or xTest or yTrain or yTest? If yTrain and/or yTest are to be scaled, then the prices predicted would be between 0 and 1 which is not very useful. How do I then use the model to predict actual prices?

I have been trying for a few days now, but I can't find a decent explanation on what part of a data is actually scaled and why?

$\endgroup$

1 Answer 1

1
$\begingroup$
  • First of all, you need scaling when the models are linear. For e.g. Linear Regression or Logistic Regression, NN. Now from your question, I understand that, you know about what scaling is.

  • Second of all, you never scale the target variable. You always scale training data and then transform it over the testing or validation data.

  • Third, scaling does improve the score in case of linear models. But in non-linear models like Decision tree or Random Forest, you don't need to do scaling.

  • Lastly, why scaling needed. To answer that, I can say that, linear models needs their data to be in Gaussian format, or in normal format. Also, it needs data with no outliers etc. And these requirements can be fulfilled by using scaling. And that's why we use scaling.

  • You can use the following:

    from sklearn.preprocessing import MinMaxScalar, RobustScalar, StandardScalar

    scalar = MinMaxScalar()

    scalar.fit(xTrain)

    xTrain = scalar.transform(xTrain)

    xTest = scalar.transform(xTest)

$\endgroup$
2
  • $\begingroup$ Thanks. I can't upvote due to my low reputation. My teacher told me I need to scale my data but then he went to a different country and I have been stuck on this since. $\endgroup$ Commented Jun 22, 2023 at 8:24
  • $\begingroup$ Happy to help...! $\endgroup$ Commented Jun 22, 2023 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.