0
$\begingroup$

My dataset contains the following two features: "movie duration" (minutes) and "tv shows duration" (seasons). If a certain sample is of type "movie", it's duration will appear under the "movie duration" feature while it's "tv show duration" feature will be zero.

Now the "movies duration" is distributed normally and the "tv show duration" is exponentially distributed. In order to scale them I used sklearn.preprocessing.MinMaxScaler for the exponential distribution of "tv show duration" and sklearn.preprocessing.StandardScaler for the normally distributed "movie duration" feature. The problem is that when I apply both scalers, the 0's in the "movie duration" feature obtain a non-zero value (which does not happen for the "tv show duration" feature) Thus, the tv shows contain values in both features while the movies have values only in the "movie duration" feature.

Example: this-

'type'    'movie duration'  'tv show duration'

movie           103                0
tv show          0                 1

Becomes this-

'type'    'movie duration'  'tv show duration'

movie      0.6331993973            0
tv show    -1.333985768        0.0588235294

First, I'm wondering if that's an actual problem (cause it may create faux dependency between the features). I had a thought to use only the nonzero values of the distribution for the standard scaler and then manually insert the zeros back, but that can also create faux dependency since the resulted distribution is between -1 to 1.

Can you think of another solution?

Note: This is a part of an unsupervised algo. (probably k-means) that will be used to create features for a supervised classification (NN).

$\endgroup$

1 Answer 1

0
$\begingroup$

There are multiple remarks to make:

Faux Dependencies

I am not sure what faux dependencies you are expecting to appear. Such a faux dependency would somehow have to treat 0 different from other values.

Keep in mind that for many algorithms a feature value of 0 is just another value. For sure it is for the two algorithms you mentioned (K-Means and NN).

The effect on NN and K-Means

Both NN and K-Means are based on distance measures. To understand the effect of the preprocessing on these algorithms, we start with the two features, let's say $x=(x_1,x_2)^T$. Both, MinMaxSclaer and StandardScaler are affine transformations, that is: $$x_i\mapsto r_ix_i+s_1\quad(i=1,2)$$ So the transformed input $\tilde{x}$ is given by $$\tilde{x} = Rx+s = (r_1x_1+s_1,r_2x_2+s_2)^T,\qquad R=\left(\begin{array}{cc}r_1 & 0\\ 0 & r_2\end{array}\right)$$ So the euclidean distance between two samples $X^A$ and $x^B$ is given by: $$d(\tilde{x}^A,\tilde{x}^B)=\|\tilde{x}^A-\tilde{x}^B\|_2=\|R(x^A-x^B)\|_2=\sqrt{r_1^2(x_1^A-x_1^B)^2+r_2^2(x_2^A-x_2^B)^2}$$ So basically you are doing a reweighting of the features. As you can see, the offsets $s_1$ and $s_2$, which cause $0$ to be mapped to some other value, have zero impact. Yet, min-max-scaling has a stronger shrinking effect than standard scaling. So effectively, you are putting more weight on "tv show duration" by using these different scalers.

Suggestion

I would suggest to use the same scaler for both features. Here, standard scaler is more robust towards outliers, so I would go for that one.

By doing so, the euclidean distance over the transformed features would be equal to an Mahalanobis distance over the original features (assuming that there is no correlation between these two features). Of course you could skip the transformation and directly run NN and K-Means with the Mahalanobis distance instead of the euclidean.

Some further remarks
  • I assume that you are only approximating "movies duration" with a normal distribution. Since the probability for negative duration-values would be larger than zero, the real distribution would be a bit different.
  • StandardScaler will not result in values between -1 and 1, but will normalize the variance to 1. One might still see values of arbitrary size.
  • Other algorithms (e.g. gaussian mixture models) also depend on the difference between two points and would follow the same argumentation. For many more, an affine transformation of the input features has mainly and effect on the speed of convergence.
  • Also for other distances (e.g. based on Manhattan norm, p-norm, maximum norm), the result would be the same.
$\endgroup$
2
  • $\begingroup$ So I was mentioning "faux dependencies" because of two reasons: 1. Having a value for the 'movie duration' when it's a 'tv show' would create a connection between then if I'm building a NN for example. $\endgroup$
    – Shir
    Jun 28, 2023 at 13:44
  • $\begingroup$ I just realized, that NN means neural network and not nearest neighbor for you. I can update my answer for neural networks. Basically, my former statement holds: zero is not that special. For a properly trained NN, it will make no big difference, whether zero, or another value is used to represent "not applicable". $\endgroup$
    – Broele
    Jun 28, 2023 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.