New answers tagged


I think you want $$S_{n}=S_{n−1}+ (x_{n}−μ_{n−1})(x_{n}−μ_{n})$$ where S, x and μ are respectively the variance, value and mean. See for explanation and derivation.


You're also scaling $y$, then of course you are getting lower error. That question was regarding scaling $X$. The same model will have very different error metrics when units on $y$ are changed: if I multiply all $y$ values by 100, the error will be 100 times larger, if I divide all $y$ values by 100 the error will be divided by 100.


Scaling doesn't affect the performance of any tree-based method, not for lightgbm, xgboost, catboost or even a decision tree. This post that elaborates on the topic, but mainly the issue is that decision trees split the feature space based on binary decisions like "is this feature bigger than this value?", and if you scale your data, the decisions ...


We scale data because certain algorithm will not work optimally esp. Gradient descent. You may check the internet for further detail. Coming to the exact question - I am assuming, you have done the analysis that the Oridinal feature will be used as a Continuous(Not Categorical) feature with its respective values. So, I will ignore this point Models don't see ...


In real life problems you may not not have the actual “test data” to fit on, for example in some time series forecasting problems. My recommendation if you want to keep safe and avoid data leakage is to fit on training data and transform on test.


If it is a classification problem, then you will use sigmoid or softmax to make the output value in (0,1) and all the value must sum to 1 as per the rule of probability.


You should not use Label Encoding for Categorical data unless there is a known ranking and that also in the specified ratio between the level values. In this case, the model will assume 10 as 2 times of 5. One-hot will add a lot of dimensions as I can see in your data. You must try other Categorical encoding techniques esp. Sum Coding Or Helmert. You ...

Top 50 recent answers are included