2
$\begingroup$

Lets take a simple database with 3 columns called x1, x2 and label for example

label is being labeled by this condition if x1-x2> 0 then label = 1 else 0 , i.e if x1 greater than x2, label is 1 else 0

x1  x2  label

100 1   True
200 2   True
4   6   False
3   10  False
500 15  True
600 20  True

I want my ML model to accurately predict label based on if x1>x2

After scaling the data with standard scaler and minmax scaler column wise, I get this data

standard= StandardScaler()
normal= MinMaxScaler()

scaled_data = standard.fit_transform(normal.fit_transform(reshaped_data))
print(scaled_data)
   x1               x2  
-0.573437   -1.171080   
-0.147090   -1.024695   
-0.982731   -0.439155   
-0.986994   0.146385    
1.131953    0.878310    
1.558300    1.610235    

After scaling I lose the relationship between x1 and x2 for example in the last data row, where 600>20 got scaled down to 1.5583 < 1.610235 which will hinder the performance of the model to learn the condition ( if x1-x2> 0 then label = 1 else 0)

Above, was just a small example to show that scaling using normal methods disturbs the relationship between columns

In my original timeseries dataset I have 200+ columns and I want to train an LSTM Model for multiclass classification.

Most of the timeseries columns are not contributing towards the label, some columns are in range 0-3000 and some columns are percentages between 0-1 and some columns are binary.

And the Labels are being labeled by some complex formula where columns are interacting with each other, being compared with each other to get the label.

It is certain that I will have to scale the data since some columns are in range 0-3000. But i want to scale the data in such a way that even after scaling, when the same formula is applied to scaled data, I get the same labels. This way I can be certain that the LSTM model will be able to learn the underlying formula and predict the label accurately.

If I scale the data normally, i.e. the relationships between the columns are disturbed, then my LSTM model performs poorly on unseen data.

How should i scale my data such that the relationship between columns are maintained even after scaling??

This is what I have tried :-

Scaling those columns together which are of the same units, for example x1,x2 should be scaled together because they both have same unit. Even if x1 has range of 0-40 and x2 has range of 0-3000, since the units are same, they should be scaled together.

Not scaling percentage columns at all, since those columns are already in range of 0-1, no point in scaling them

scaling all other columns independently.

What I achieved:-

with the above method I got the result I want (99% accuracy on unseen data), however I had to do a lot of hard coding since there are 200+ columns, I had to go through each of them and scale them separately.

is there an easier way of doing this?

$\endgroup$
2
  • $\begingroup$ (Applying MinMaxScaler before StandardScaler is redundant.) $\endgroup$
    – Ben Reiniger
    Commented Feb 26 at 20:34
  • $\begingroup$ Yes I will keep that in mind. However, I tried all the combinations of scaling techniques (first standardizing it, then normalizing it) and none of them gave me good accuracy on unseen data because all the scaling techniques removes any relationship I have between my columns. $\endgroup$ Commented Feb 27 at 2:02

4 Answers 4

2
$\begingroup$

Not a complete answer, but some clarifications to help frame things.

Models that are arithmetically based can still produce your relationship: if the scaling is $\tilde{x}_i := (x_i - b_i)/m_i$, then $x_1 - x_2 > 0$ is equivalent to $m_1 \tilde{x}_1 - m_2 \tilde{x}_2 + b_1 - b_2 > 0$. This is just another linear relationship between the new $\tilde{x}_i$, so linear regressions or neural networks (with intercepts/biases) should be fine at finding it.

Models with regularization will care; that's one of the reasons we scale: to put regularization penalties on the same scale. But in your example, where $x_2$ has a much smaller scale but your relationship is still just $x_1 > x_2$, this may cause some problems: $m_1 \gg m_2$, and so the regularization penalty will apply differently on the estimated coefficients of $x_1$ and $x_2$, which I don't think is desirable.

Tree-based models (the usual ones anyway) won't care at all: the new relationship looks exactly the same to a tree; they don't care about scaling, only relative ordering. (The original relationship isn't trivial for a tree to approximate though.)

Finally, the actual fitting procedure might care: the backpropagation in neural networks tends to work best when parameters are roughly in the same range, so again in your example with $m_1\gg m_2$, the scaled data might actually be harder to learn from.

$\endgroup$
1
$\begingroup$

Yes, don't scale your data.

Not all binary classification models (ex: Tree-based models) need the data to be scaled, and based on your data, these are the most potentially applicable kinds of models.

$\endgroup$
3
  • $\begingroup$ I just edited my question. My data is time series and I want to perform multiclass classification using LSTM. So I assume, I will have to scale the data. Or even in that case I would be ok without scaling? $\endgroup$ Commented Feb 26 at 17:21
  • $\begingroup$ Even using LSTM you don't necessarily need to scale your data. $\endgroup$
    – m13op22
    Commented Feb 26 at 17:23
  • 1
    $\begingroup$ Hmmm. I will train my data without scaling anything and see whether it will give me good results. I assumed I will have to scale my data since some columns ranges between 0-3000 because I read online that NNs have a tendency to skew the result if the inputs have large values. $\endgroup$ Commented Feb 26 at 17:29
1
$\begingroup$

How are you applying the scaling? If you do the following,

std_scaler = StandardScaler()
minmax_scaler = MinMaxScaler()
df_std = std_scaler.fit_transform(df)
df_min_max = minmax_scaler.fit_transform(df)

The results ofpd.DataFrame(df_std):

0 1 2
0 -0.573437 -1.171080 0.707107
1 -0.147090 -1.024695 0.707107
2 -0.982731 -0.439155 -1.414214
3 -0.986994 0.146385 -1.414214
4 1.131953 0.878310 0.707107
5 1.558300 1.610235 0.707107

or pd.DataFrame(df_min_max):

0 1 2
0 0.162479 0.000000
1 0.329983 0.052632
2 0.001675 0.263158
3 0.000000 0.473684
4 0.832496 0.736842
5 1.000000 1.000000

Min max scaling subtracts the minimum and divides by max-min. Standard scaling subtracts the mean and divides by the std deviation. In both cases, the relationship x1 > x2 is preserved. I assume your use case is more complicated than this, if so, please let us know. Otherwise you don't really don't need machine learning, indeed you don't need to scale your data at all if it's just x1 > x2 that you're interested in computing.

$\endgroup$
2
  • $\begingroup$ I just edited my question. When testing with the example data, I performed standard.fit_transform(normal.fit_transform(reshaped_data)) that is, first applied minmax then applied standard scaler. Even if I applied just one those, there will be cases where the relationship between columns will change. $\endgroup$ Commented Feb 26 at 17:22
  • $\begingroup$ Also the formula I am trying to learn is not just x1>x2. It is if (x1>x2) and ((x1-x2)/x3) >x4: then label =1. this is the condition just for one label, I have different conditions for my 5 different labels. $\endgroup$ Commented Feb 26 at 17:26
1
$\begingroup$

You already have the perfect predictive model: if x1-x2> 0 then label = 1 else 0. There is no machine learning to do. Just label according to this rule. If you, for whatever reason, have to scale the variables, then just crunch through the algebra. For scaling transformations $T_1$ and $T_2$, define new, scaled variables by $s_1 = T_1(x_1)$ and $s_2 = T_2(x_2).$

$$ x_1-x_2>0\\ \Updownarrow\\ T^{-1}_1(s_1)-T^{-1}_2(s_2)>0 $$

You now have the condition to give labels in terms of the scales values, $s_1$ and $s_2,$ instead of in terms of $x_1$ and $x_2.$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.