If I am trying to build a classification model where Values for columns have different ranges: For Example: Column A in training set ranges from 0 - 30 but for testing set it ranges from 0 - 80.
Would a classification model still work?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Welcome Aboard, you will do some Preprocessing (like normalisation) on both the train and test sets, so it won't matter but yes this isn't expected. (I may be wrong, still trying to understand ML) $\endgroup$– AdityaApr 28, 2018 at 0:52
-
$\begingroup$ If the dataset does not specify a split for training and testing, I would recommend that you create or own splits which sample from both ranges roughly equally, so you are getting a true sampling of the range of data (unless these are discrete values which correspond to composing categorical variables, and you want to test how well the model does these compositions). $\endgroup$– Pavel SavineApr 28, 2018 at 3:29
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
On principle, you should try to achieve a fairly balanced distribution between your training and testing sets. Otherwise your model might not make appropriate generalizations. Additionally, you should use some form of normalization such as feature scaling.
It is difficult (or even impossible) to understand your data distribution by simply looking at the range. It would be better to look at the mean/median or even a histogram/normal curve.
Let's say, for example, that our ordered column A in your training/testing sets look as such (before normalization):
\begin{array}{|c|} \hline &column \ A \ split\\ \hline training && testing \\ 0 && 0\\ 4 && 2\\ 5 && 3\\ 7 && 3\\ 9 && 8\\ 15 && 11\\ 23 && 15\\ 30 && 80\\ \hline \end{array}
\begin{array}{|c|} \hline &set\ metrics\\ \hline & training & testing\\ range & [0,30] & [0,80]\\ mean & 11.625 & 15.250\\ median & 8.0 & 5.5\\ stddev & 10.309 & 26.650\\ \hline \end{array}
\begin{align} \large\frac{1}{\sqrt{2\pi}\cdot\sigma}e^{\frac{-(x-\mu)^2}{2\sigma^2}} \end{align}
(red=training, blue=testing)
Here we can see that despite the large range discrepancy, our sets are relatively balanced.
We can observe a problematic training/testing split in this example where we shift the training mean from 11.625 to 100:
Moreover, we could generate another hypothetical dataset with a bad training/testing split where our set ranges are identical.
TL;DR:
Do not evaluate the set distribution by range; instead use mean/median or histogram/normal curve. Also normalize your data.
$\begingroup$
$\endgroup$
The range of the dataset values can be influenced by extreme values (outliers), so it is not the right way to check if the training set and the test set follow the same distribution.
First of all you need to make a comparison using the five-number summary (min-Q1-median-Q3-max) and make a first conclusion for the distribution equality.
A second option could be to perform a statistical test as the Kolmogorov-Smirnov test to check for distribution equality and see if there is a significant difference or not between the two samples.
If the is a significant difference, then the classifier trained using a dataset with different distribution from the test set will have a poor performance on the test set. A solution could be to use Transfer Learning/Domain Adaptation Methods to tackle this difference and adapt the classifier trained on a different distribution for the test set instances.
answered May 1, 2018 at 21:54