Can Neural Networks be trained to smooth values / output the average?

Question

Let's say we have a neural network with one input neuron and one output neuron. The training data $(x, f(x))$ is generated by a process

$$f(x) = ax + \mathcal{N}(b, c)$$

with $a, b, c \in \mathbb{R}^+$, e.g. something like

feature  | target
-----------------
0             0.0
0             1.0
0             1.5
0            -1.2
0            -0.9
...

I know that neural networks can deal pretty well with labeling errors in classification problems. Meaning if you have a large dataset and a couple of examples have the wrong label, they get basically ignored.

But for this kind of problem I'm not too sure. A first experiment indicates that they do smooth values.

Are there choices in architecture / training which help the smoothing / averaging / removal of noise?

What I tried

I created a network which can solve this kind of regression problem without noise. It gets a MSE of about 0.0005. When I add a bit of noise to the training set only, I get an MSE of 0.001:

#!/usr/bin/env python

# core modules
import random

# 3rd party modules
from keras.models import Sequential
from keras.layers import Dense
from sklearn.model_selection import train_test_split
import numpy as np


def main(add_noise=True):
    # Get data
    xs, ys = create_data_points(10000)
    x_train, x_test, y_train, y_test = train_test_split(xs, ys, test_size=0.20)

    # Add noise to training data
    if add_noise:
        noise = np.random.normal(0, 0.1, len(x_train))
        x_train = x_train + noise

    # Create model
    model = create_model()
    model.compile(optimizer='rmsprop',
                  loss='mse',
                  metrics=['mse'])

    # Fit model to data.
    model.fit(x_train, y_train, epochs=10, batch_size=32, verbose=1)

    # Evaluate
    y_pred = model.predict(x_test, batch_size=100).flatten()
    print("MSE on test set:")
    print(((y_pred - y_test)**2).sum() / len(y_test))


def create_data_points(nb_points):
    xs = []
    ys = []
    for i in range(nb_points):
        x = random.random()
        xs.append(x)
        ys.append(2 * x)
    return np.array(xs), np.array(ys)


def create_model(input_dim=1, output_dim=1):
    model = Sequential()
    model.add(Dense(200, input_dim=input_dim, activation='relu'))
    model.add(Dense(200, input_dim=input_dim, activation='relu'))
    model.add(Dense(output_dim, activation='linear'))
    return model


if __name__ == '__main__':
    main()

Outliers

In an earlier version of this question I wrote "outlier" when I meant "label noise". For outliers, there is:

• The Effects of Outliers Data on Neural Network Performance

Answer 1

In general simpler models are more robust to noise in the input. The strength of neural networks is also their biggest 'gotcha' - they are extremely expressive. This means they easily overfit and can be sensitive to noise in inputs. The strategy of simplifying a model to make it more robust to noise is called regularization. There are many types:

• use smaller hidden layers (i.e. 20 instead of 200 nodes)
• use dropout, where during training inputs are randomly set to 0 during training (makes the network more robust to noise overall)
• stop training earlier - 'early stopping'
• use L1 or L2 regularization, which imposes a cost on the weights

All of these can be done from with Keras. You want to make your network more robust to noise without decreasing your validation quality. To do this, I would try the above suggestions in order. You can measure overfitting by looking at the difference between predictive accuracy on your training data and validation data. If they are very different - your model has learned structure in your training data that is not in your validation that is NOT what you want. Try to fiddle regularization nobs until (1) your train-validation AUC or accuracy is very similar, (2) your validation AUC/accuracy is still sufficiently high.

Answer 2

I know that neural networks can deal pretty well with outliers.

Not necessarily.

It depends on your loss function and sample size.

In linear regression, the loss function is MSE. MSE is not robust against outliers, so linear regression is not robust against outliers. Having larger sample will mitigate the impact of a outliers.

Same principles applies to neural network.

Are there choices in architecture / training which help the smoothing / averaging / removal of noise?

That's just the problem of over fitting.

@tom has covered the most commonly used techniques. The only thing I will add is using larger sample size. Sometime this can be achieve cheaply by using data augmentation techniques (i.e. image rotation etc).