Hello I'm studying a classification problem with KNN right now.

I have many numeric features that I normalized with MinMaxScaler, I also got some OHE categorical features that not seem to cause the drop, and with this method my accuracy dropped almost 10%.

I used holdout method and cross validation for the results.

What are some follow up actions I could use to try to up my accuracy?

Should I try another normalization technique? Should I try another model ? Anything else?


1 Answer 1


Is bad luck.

Look what (I think) is happening:

When you use MinMaxScaler, what you do is relativize (reduce or augment) the distance between individuals in a way it was not before the change.

Let's suppose your best model is defined by the variable $X_j$ from the variable set $X=\{X_1,X_2,...,X_n\}$. When you change the set to an scalled one: $S=\{S_1,S_2,...,S_n\}$, what you do is giving the same importance to every variable, instead of preserving what the variables said before scalling.

$X_3$ gives better information on the output than $S_3$ (and in $S$, all are scalled), $S_3$ gives less information because is mixed with all other variables.

Imagine $X_3$ as something like "age" (and you are determining probability of having cancer), the greater the age, the greater the probability. This is true for $S_3$ also, but when $S_3$ is combined with the rest of $S$, $S_3$ looses importance amongst them. With $X_3$, age keeps its relative importance (bigger values amongst lesser values).

This does not happen very often, you could find yourself on the opposite situation: When combined $S$ might be a very powerful set.

That is why I think you were simply, unlucky.

  • $\begingroup$ So should I use another normalization technique or that's not gonna work again? $\endgroup$
    – dungeon
    Apr 22, 2019 at 19:00
  • $\begingroup$ It might work, try another normalization technique and observe what happens. Try to use trees too. $\endgroup$ Apr 22, 2019 at 19:02
  • $\begingroup$ Can you be more specific about what you mean about trees? I'm just learning is this just a different model (decision tree model?) I'm planning to use many models on this problem but I was trying to get better results from knn $\endgroup$
    – dungeon
    Apr 23, 2019 at 16:35
  • $\begingroup$ The trees rely on intervals when working with numeric variables, so for trees is the same to work with $X$ than to work with $S$, because the separations will be the same. If you work with scalled variables there is also no interaction. $\endgroup$ Apr 23, 2019 at 20:22

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .