1
$\begingroup$

I am interested in a discussion in encoding and scaling categorical features, notably imbalanced categorical features. The context is neural networks (gbdts should handle this easily). It is known that numerical values should be rescaled with a mean near zero and a covariance constant (1?) since at least Lecun 98.

However for rare event this mean it will bring unusually high values.

Typically, having a 1 in 1000 features:

x = np.random.random(1000000)>0.999
np.unique(((x-x.mean())/x.std()))

imply having two values after rescaling:

array([-3.01134784e-02,  3.32077214e+01])

That is 0.03 and 30. Is this the correct way to go? Are there any strong source on this?

Improve this question
$\endgroup$
7
  • $\begingroup$ Scaling for categorical variables is already... fishy, in case of asymmetric variables you might want to use robust scaling. $\endgroup$
    – user2974951
    Commented Feb 2 at 11:14
  • $\begingroup$ What do you mean by fishy? robust scaling? (because the robust scaling I know, from sklearn would give nans). $\endgroup$
    – Lucas Morin
    Commented Feb 2 at 11:15
  • 1
    $\begingroup$ See Should you ever standardise binary variables? and Compare the effect of different scalers on data with outliers. $\endgroup$
    – user2974951
    Commented Feb 2 at 11:20
  • $\begingroup$ They don't really provide a robust answer on what should be done and why. Only practical answer makes the assumption of balanced data. Regarding the robust scaling (that apply to continuous features) would not work here as 1st and 3rd quantile are equals. $\endgroup$
    – Lucas Morin
    Commented Feb 2 at 13:12
  • 1
    $\begingroup$ See also datascience.stackexchange.com/q/31652/55122, datascience.stackexchange.com/q/80234/55122, datascience.stackexchange.com/q/56444/55122. The first one comes closest to an answer here: Neil Slater reports slight improvements in neural networks after scaling dummy variables (which will be imbalanced when there are several levels in the original categorical). With neural networks, it won't make a difference in the actual functional space unless you apply regularization; but it may well make a difference numerically while fitting. $\endgroup$
    – Ben Reiniger ♦
    Commented Feb 2 at 15:19

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.