Hello I'm studying a classification problem with KNN right now.

I have many numeric features that I normalized with MinMaxScaler, I also got some OHE categorical features that not seem to cause the drop, and with this method my accuracy dropped almost 10%.

I used holdout method and cross validation for the results.

What are some follow up actions I could use to try to up my accuracy?

Should I try another normalization technique? Should I try another model ? Anything else?


Is bad luck.

Look what (I think) is happening:

When you use MinMaxScaler, what you do is relativize (reduce or augment) the distance between individuals in a way it was not before the change.

Let's suppose your best model is defined by the variable $X_j$ from the variable set $X=\{X_1,X_2,...,X_n\}$. When you change the set to an scalled one: $S=\{S_1,S_2,...,S_n\}$, what you do is giving the same importance to every variable, instead of preserving what the variables said before scalling.

$X_3$ gives better information on the output than $S_3$ (and in $S$, all are scalled), $S_3$ gives less information because is mixed with all other variables.

Imagine $X_3$ as something like "age" (and you are determining probability of having cancer), the greater the age, the greater the probability. This is true for $S_3$ also, but when $S_3$ is combined with the rest of $S$, $S_3$ looses importance amongst them. With $X_3$, age keeps its relative importance (bigger values amongst lesser values).

This does not happen very often, you could find yourself on the opposite situation: When combined $S$ might be a very powerful set.

That is why I think you were simply, unlucky.

  • $\begingroup$ So should I use another normalization technique or that's not gonna work again? $\endgroup$ – dungeon Apr 22 '19 at 19:00
  • $\begingroup$ It might work, try another normalization technique and observe what happens. Try to use trees too. $\endgroup$ – Juan Esteban de la Calle Apr 22 '19 at 19:02
  • $\begingroup$ Can you be more specific about what you mean about trees? I'm just learning is this just a different model (decision tree model?) I'm planning to use many models on this problem but I was trying to get better results from knn $\endgroup$ – dungeon Apr 23 '19 at 16:35
  • $\begingroup$ The trees rely on intervals when working with numeric variables, so for trees is the same to work with $X$ than to work with $S$, because the separations will be the same. If you work with scalled variables there is also no interaction. $\endgroup$ – Juan Esteban de la Calle Apr 23 '19 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.