35
$\begingroup$

What are the pitfalls of doing so and why is it a bad practice? Is it possible that the model starts to learn the images "by heart" instead of understanding the underlying logic?

$\endgroup$
1

9 Answers 9

42
$\begingroup$

Yes, you put it quite correctly.

As a teacher, you wouldn’t give your students an exam that’s got the exact same exercises you have provided as homework: you want to find out whether they (a) have actually understood the intuition behind the methods you taught them and (b) make sure they haven’t just memorised the homework exercises.

$\endgroup$
8
  • 27
    $\begingroup$ An infamous example I've seen was a classifier designed to detect fish. The training dataset had many pictures of a fisherman holding a fish versus not holding a fish. The classifier seemed to work perfectly on the training data, but later failed to detect any fish under any other conditions. It turned out the neural network focused solely on the fingers of the fisherman, because they were in a consistent position every time he held a fish. $\endgroup$
    – vsz
    Dec 14, 2020 at 7:07
  • 16
    $\begingroup$ Another one: A classifier to detect enemy tanks ... worked perfectly in training, but effectively only learned that (pictures of) friendly tanks were shot during the day, enemy tanks were shot at night. $\endgroup$
    – fho
    Dec 14, 2020 at 14:01
  • 15
    $\begingroup$ I remember reading one about telling wolfs apart from dogs, and the model learned that wolves had snow in the background, not that wolves look like wolves. $\endgroup$
    – Dave
    Dec 14, 2020 at 15:52
  • 6
    $\begingroup$ @Dave That one is often misremembered: the researchers were well aware of the snow=wolves (and grass = dog), as it was an intentional flaw they put into the training data set. They weren't building a model to tell wolves from dogs, though: they were looking to see how much (certain types of) people would retain their trust in the (trained) algorithm's soundness after watching it perform. It was a psychological experiment. $\endgroup$ Dec 15, 2020 at 11:08
  • 8
    $\begingroup$ None of the three examples offered in the comments so far are good examples of training and testing the model on the same data. They're examples of testing models on data that is substantially different to training, but if you know that sort of data exists you should include some in the training set! A model trained on objects in multiple settings can still overfit if you test it on exactly those images. It might not be as simple as snow/no snow but the color of a couple of pixels is all you need for a perfect score if you never have to care about out-of-sample classification. $\endgroup$
    – Will
    Dec 15, 2020 at 12:51
13
$\begingroup$

It is wrong because:

  • it is fundamentally incorrect (a theoretical concern)
  • it leads to bad results (a practical concern)

It is fundamentally incorrect because usually the objective of testing a model is to estimate how well it will perform predictions on data that the model didn't see.

It's quite hard to come up with good estimates of real-world performance, even when you do everything correctly. If you use training data to estimate the performance the result is worse than useless, it's actively misleading.

There's several ways that doing this can lead to bad results.

Overfitting

If you're training a complex model with small amount of data, your model is very likely to overfit. In a simplified way, we can say that if the has a lot of "memory" (parameters), it memorizes the training data, and fails to understand its underlying structure.

Imagine that you're building a model that predicts house price based on the floor area. Your training set looks like this:

area  price
30    100001
50    150002
80    200003

You train your model, then ask it to predict the price for a house of area=50, and it tells you that the price should be 150002. Is that impressively accurate? Not really. It's just memorizing the training data.

Overfitting is commonly detected through a large difference in performance between the training and test set. If you test on the training set, you're unable to detect overfitting.

Concept drift

If you make sure you're training a very simple model on a large amount of data, even if there's no overfitting, it's common for models to suffer from concept drift.

This basically means that the underlying structure of the data can change over time. For example, trying to predict how many sales a store is going to make on the weekend after training on data from Monday to Friday.

If your test data is not diverse enough along the time dimension vs the training set, you won't catch that problem.

$\endgroup$
1
  • 1
    $\begingroup$ Exactly. It is fundamentally incorrect. It is mal-implementation of statistics. $\endgroup$
    – Stian
    Dec 14, 2020 at 14:12
8
$\begingroup$

It can happen that the model you train learns "too much" or memorizes the training data, and then it performs poorly on unseen data. This is called "overfitting".

The problem of training and testing on the same dataset is that you won't realize that your model is overfitting, because the performance of your model on the test set is good. The purpose of testing on data that has not been seen during training is to allow you to properly evaluate whether overfitting is happening.

$\endgroup$
9
  • 2
    $\begingroup$ There is something called "data leakage" that consists of having part of your training data inadvertently leaked to the validation/test set. This can give you a false evaluation of overfitting, as you will think that there is no overfitting, but due to the overlap, you may be having overfitting without realizing it. This is why it is important to properly avoid overlaps between the training and test data. $\endgroup$
    – noe
    Dec 13, 2020 at 14:49
  • 2
    $\begingroup$ And overfitting can just be detected with a separate test set, not avoided. In order to avoid overfitting there are techniques like "regularization". $\endgroup$
    – noe
    Dec 13, 2020 at 14:50
  • 2
    $\begingroup$ Just to make it clear - the problem with overfitting is not just that the model becomes too good on the training data, it's (also) that it gets better on the training data but worse on other data. $\endgroup$
    – kutschkem
    Dec 14, 2020 at 16:06
  • 1
    $\begingroup$ @ncasas Regarding data leakage: I once created a code recommendation engine based on the name of the function in which the programmer is currently writing. The idea was inside getXXX methods calling other getXXX methods will be more likely than setXXX. Worked surprisingly well, but the big catch is that in my training (and test!) data, a very big proportion was actually generated code (from ANTLR I think). So I am suspicious the model just learned how ANTLR-generated code looks but I didn't have time to clean up the data and try again, so I will never know for sure. $\endgroup$
    – kutschkem
    Dec 14, 2020 at 16:14
  • 1
    $\begingroup$ @ncasas Another example: I worked in a project classifying Wikipedia articles, and it was important to exclude the comment section from the features because otherwise we would be training on discussions about the issues we were trying to detect. $\endgroup$
    – kutschkem
    Dec 14, 2020 at 16:17
6
$\begingroup$

Simple answer: circular reasoning. The fact that your model "knows" the answer to something you've already told it the answer to really doesn't prove anything.

Put another way: the entire point of testing is to get some sense of how well your model would do with data it hasn't seen yet, and testing it with data that it has already seen doesn't do that.

$\endgroup$
3
$\begingroup$

Is it possible that the model starts to learn the images by heart instead of understanding the underlying logic?

If the model memorizes the training data when that same data is used for the "test" set, it would still memorize the training data when different data was used for the "test" set. Using a separate "test" set cannot prevent that memorization from happening. More generally, the "test" set has no direct impact on model training.

However, the separate "test" set does allow the researchers to identify that the model is indeed memorizing the individual data samples and targets instead learning the underlying patterns. When the researchers see the loss decreasing on the training set but increasing on the "test" set, they know this overfitting is taking place. At that point they can tune the model's hyperparameters, specifically trying to lower the model's capacity (i.e. number of nodes and/or layers), and then retrain the model to see if the issue has been resolved.

Because the researchers use the "test" set to tune the model's hyperparameters, the "test" data can have an indirect impact on the final model's performance. It could be that the researchers pick hyperparameters that work well for "test" set, but not for the data in general. For that reason, it is sometimes recommended to use 3 distinct data sets: the training set used to train the model, the initial "test" set used to address overfitting and other issues by tuning hyperparameters (this is more commonly known as the validation set), and a final test set which is only used to evaluate the finalized model (and has no impact, direct or indirect, on model training).

$\endgroup$
2
$\begingroup$

To express it in a different way, that might be more useful when explaining to impatient stakeholders:

Imagine that you go to a travelling fair and a lady with many shawls and a crystal ball tells you, "I can look at a person and tell them if they are married or not." You are not sure if this is for real.

If she starts pointing at her colleagues from the fair and tells you, "he is married, she isn't, the other woman also isn't" - what does this tell you? Nothing. She already knows these people, she knows who is married! To start trusting her ability, you want her to make her guesses about people she's never seen.

In data science, you always have the problem whether people (including you!) should trust the model or not. It can prove itself by showing that it can find information which it didn't know beforehand. It has to know its training data by definition, so your only option is to keep some data "hidden" from it (the test set).

In fact, it is ideal that, if you suspect your data is too uniform, to do a second testing with a different dataset created in a different way, to confirm it is working in general. This is done mostly in science, if data is available, e.g. if you trained and tested data on patients from one hospital, you ideally try it on patients from a different hospital, just in case data was coded differently, or you had selection bias or whatever.

$\endgroup$
2
$\begingroup$

To give a simple illustration of how bad overfitting can be, consider the example of fitting (training) a polynomial of order equal to the number of points of data you have. In this case I've generated data with a slope and some normally distributed random noise added. If you test it with exactly the same x & y values that you used to generate the polynomial fit by looking at the residuals, all you see is the numerical error, and you might naively say it's a good fit, or at least better than the linear fit (plotted in green) which has much larger residuals. If you plot the actual polynomial you get (in red), you'll probably see that it actually does a terrible job of interpolating between this test data (since we know that the underlying process is simply a straight line), like so:

enter image description here

If you generate a new set of data with the same x-values, you see that as well as failing at interpolating, this performs about the same as the linear fit in terms of residuals:

enter image description here

And perhaps worst of all, it fails spectacularly when attempting to extrapolate as the polynomial predictably blows up in both directions:

enter image description here

So if "prediction" for your model is interpolating, then overfitting makes it bad at that and won't be detected unless you test it on non-training data. If prediction is extrapolating, then most likely it's even worse at that than it is at interpolating, and again you won't be able to tell unless you test it on the right kind of data.

Python code used to generate these plots:

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(0)

nSamples = 15
slope = 10
xvals = np.arange(nSamples)
yvals = slope*xvals + np.random.normal(scale=slope/10.0, size=nSamples)

plt.figure(1)
plt.clf()
plt.subplot(211)
plt.title('"Perfect" polynomial fit')
plt.plot(xvals, yvals, '.', markersize=10)

polyCoeffs = np.polyfit(xvals, yvals, nSamples-1)
poly_model = np.poly1d(polyCoeffs)
linearCoeffs = np.polyfit(xvals, yvals, 1)
linear_model = np.poly1d(linearCoeffs)
xfit = np.linspace(0, nSamples-1, num=nSamples*50)
#yfit_dense = poly_model(xfit)
plt.plot(xfit, poly_model(xfit), 'r')
plt.plot(xfit, linear_model(xfit), 'g')

plt.subplot(212)
plt.plot(xvals, poly_model(xvals) - yvals, 'r.')
plt.plot(xvals, linear_model(xvals) - yvals, 'g.')
plt.title('Fit residuals for training data (nonzero only due to numerical error)')

#%% Testing interpolation
plt.figure(2)
plt.clf()
test_yvals = slope*xvals + np.random.normal(scale=slope, size=nSamples)
plt.subplot(211)
plt.title('Testing "perfect" polynomial fit with new samples')
plt.plot(xvals, test_yvals, '.', markersize=10)
plt.plot(xfit, poly_model(xfit), 'r')
plt.plot(xfit, linear_model(xfit), 'g')

plt.subplot(212)
plt.title('Fit residuals for test data')
plt.plot(xvals, poly_model(xvals) - test_yvals, 'r.')
plt.plot(xvals, linear_model(xvals) - test_yvals, 'g.')

#%% Testing extrapolation
extrap_xmin = -5
extrap_xmax = nSamples + 5
xvals_extrap = np.arange(extrap_xmin, extrap_xmax)
yvals_extrap = slope*xvals_extrap + np.random.normal(scale=slope, size=len(xvals_extrap))

plt.figure(3)
plt.clf()
plt.subplot(211)
plt.title('Testing "perfect" polynomial fit extrapolation')
plt.plot(xvals_extrap, yvals_extrap, '.', markersize=10)
plt.plot(xvals_extrap, poly_model(xvals_extrap), 'r')
plt.plot(xvals_extrap, linear_model(xvals_extrap), 'g')

plt.subplot(212)
plt.title('Fit residuals for extrapolation')
plt.plot(xvals_extrap, poly_model(xvals_extrap) - yvals_extrap, 'r.')
plt.plot(xvals_extrap, linear_model(xvals_extrap) - yvals_extrap, 'g.')
$\endgroup$
1
$\begingroup$

In addition to all the good explanation about overfitting I would also quote the Goodhart's law. Since you trained by optimizing the train loss (some kind of metric) it is has thus become a bad metric to measure the quality of your model.

$\endgroup$
3
  • 1
    $\begingroup$ Can you explain how it has become a bad metric ? $\endgroup$ Feb 1, 2023 at 13:16
  • $\begingroup$ It would be great if you can describe Goodhart's Law and why it is relevant to the problem given in the question. $\endgroup$ Feb 3, 2023 at 2:45
  • $\begingroup$ Goodhart's Law says, "When a measure becomes a target, it ceases to be a good measure". Here during training, the loss is computed from the train samples, and this loss (performance metric) is optimised to train the model. Since it has been optimised, it has thus become a biased performance metric. Hence computing a metric from the train samples will lead to creating a biased metric. $\endgroup$
    – Xmaster6y
    Feb 3, 2023 at 15:47
-1
$\begingroup$

There is nothing wrong with testing the model on data you trained on, but there is something wrong with not testing your model on data it has not seen before.

As other answers have explained, tests on the data that the model was trained on are by no means substitute for tests on new data, and in case your model is overfitting these results can be very different. However, testing the model on the data it trained on is still valuable. For example, if you model does not fit the data it trained on well, then you know that you have an underfitting problem, and your model is too simple.

In other words, if your out of sample accuracy is 0.6, you would proceed differently depending on what your in sample accuracy is: if it is 0.999, then you are overfitting, if it is 0.62, then you are underfitting.

Usually you look at both, as it helps guide you model improvement direction.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.