5
$\begingroup$

I'm trying to use 2-5-1 neural network to approximate function $$x_1 \in[-3,3], x_2\in[-1,3], f(x_1,x_2)=\sin(2x_1+x_2)$$

I used code from Implementing a flexible neural network with backpropagation from scratch, to avoid using any complex libraries and tried teaching my network to approximate with following data:

# Define dataset
n = 40
np.random.seed(4)
x1_low, x1_up = -3, 3
x2_low, x2_up = -1, 3
x1s = np.random.uniform(x1_low, x1_up, size=n)
x2s = np.random.uniform(x2_low, x2_up, size=n)

Xs = []
for _x1 in x1s:
    for _x2 in x2s:
        Xs.append([_x1, _x2])

Zs = [my_func(_x1, _x2) for _x1, _x2 in Xs]

# Define test data
x1_pred = np.random.uniform(x1_low, x1_up, size=n)
x2_pred = np.random.uniform(x2_low, x2_up, size=n)
Xs_pred = []
for _x1, _x2 in zip(x1_pred, x2_pred):
    Xs_pred.append([_x1, _x2])
actual_ys = [my_func(_x1, _x2) for _x1, _x2 in Xs_pred]

# Train and test neural network
for ee in range(0, 4):
    for e in range(1, 4):
        alpha = e / 10 ** ee
        nn = NeuralNetwork()
        nn.add_layer(Layer(2, 5, 'tanh'))
        nn.add_layer(Layer(5, 1, 'sigmoid'))
        errors = nn.train(Xs, Zs, alpha, 300)
        print('Accuracy: %.2f%%' % (nn.accuracy(nn.predict(Xs_pred), actual_ys) * 100))
        # Plot changes in mse
        plt.plot(errors)
        plt.ylim([0, 1])
        plt.title(str('Changes in MSE - alpha ' + str(alpha)))
        plt.xlabel('Epoch (every 10th)')
        plt.ylabel('MSE')
        plt.show()

But I can't seem to have MSE lower than $0.4$. What can I do here to make it more accurate?

$\endgroup$
2
  • $\begingroup$ what have you tried so far? $\endgroup$
    – oW_
    May 31, 2019 at 21:22
  • $\begingroup$ I change max epochs, learning rate, and size data to learn from. I'm wondering if I have some error in my attached code, that prevents me from finding solution. All my approximations equal zero... $\endgroup$
    – Naan
    May 31, 2019 at 23:09

3 Answers 3

3
+50
$\begingroup$

One problem that I see is that notice that sine is a function that takes value from $-1$ to $1$ but sigmoid function takes value from $0$ to $1$.

Hence you are being penalized whenever the sine value takes negative value.

You might like to try to change your last layer to a tanh layer or alternatively, rather than predicting sine directly, predict $\frac{\sin(2x_1+x_2)+1}{2}$ first.

I manage to achieve MSE of $0.228686$ using the tanh modification. Of course, you can still try to tune other parameters and try other stuff to improve the model.

import numpy as np
import matplotlib.pyplot as plt
import math

class Layer:
    """
    Represents a layer (hidden or output) in our neural network.
    """

    def __init__(self, n_input, n_neurons, activation=None, weights=None, bias=None):
        """
        :param int n_input: The input size (coming from the input layer or a previous hidden layer)
        :param int n_neurons: The number of neurons in this layer.
        :param str activation: The activation function to use (if any).
        :param weights: The layer's weights.
        :param bias: The layer's bias.
        """

        self.weights = weights if weights is not None else np.random.rand(n_input, n_neurons)
        self.activation = activation
        self.bias = bias if bias is not None else np.random.rand(n_neurons)
        self.last_activation = None
        self.error = None
        self.delta = None

    def activate(self, x):
        """
        Calculates the dot product of this layer.
        :param x: The input.
        :return: The result.
        """

        r = np.dot(x, self.weights) + self.bias
        self.last_activation = self._apply_activation(r)
        return self.last_activation

    def _apply_activation(self, r):
        """
        Applies the chosen activation function (if any).
        :param r: The normal value.
        :return: The "activated" value.
        """

        # In case no activation function was chosen
        if self.activation is None:
            return r

        # tanh
        if self.activation == 'tanh':
            return np.tanh(r)

        # sigmoid
        if self.activation == 'sigmoid':
            return 1 / (1 + np.exp(-r))

        return r

    def apply_activation_derivative(self, r):
        """
        Applies the derivative of the activation function (if any).
        :param r: The normal value.
        :return: The "derived" value.
        """

        # We use 'r' directly here because its already activated, the only values that
        # are used in this function are the last activations that were saved.

        if self.activation is None:
            return r

        if self.activation == 'tanh':
            return 1 - r ** 2

        if self.activation == 'sigmoid':
            return r * (1 - r)

        return r


class NeuralNetwork:
    """
    Represents a neural network.
    """

    def __init__(self):
        self._layers = []

    def add_layer(self, layer):
        """
        Adds a layer to the neural network.
        :param Layer layer: The layer to add.
        """

        self._layers.append(layer)

    def feed_forward(self, X):
        """
        Feed forward the input through the layers.
        :param X: The input values.
        :return: The result.
        """

        for layer in self._layers:
            X = layer.activate(X)

        return X

    def predict(self, X):
        """
        Predicts a class (or classes).
        :param X: The input values.
        :return: The predictions.
        """

        ff = self.feed_forward(X)

        # One row
        if ff.ndim == 1:
            return np.argmax(ff)

        # Multiple rows
        return np.argmax(ff, axis=1)

    def backpropagation(self, X, y, learning_rate):
        """
        Performs the backward propagation algorithm and updates the layers weights.
        :param X: The input values.
        :param y: The target values.
        :param float learning_rate: The learning rate (between 0 and 1).
        """

        # Feed forward for the output
        output = self.feed_forward(X)

        # Loop over the layers backward
        for i in reversed(range(len(self._layers))):
            layer = self._layers[i]

            # If this is the output layer
            if layer == self._layers[-1]:
                layer.error = y - output
                # The output = layer.last_activation in this case
                layer.delta = layer.error * layer.apply_activation_derivative(output)
            else:
                next_layer = self._layers[i + 1]
                layer.error = np.dot(next_layer.weights, next_layer.delta)
                layer.delta = layer.error * layer.apply_activation_derivative(layer.last_activation)

        # Update the weights
        for i in range(len(self._layers)):
            layer = self._layers[i]
            # The input is either the previous layers output or X itself (for the first hidden layer)
            input_to_use = np.atleast_2d(X if i == 0 else self._layers[i - 1].last_activation)
            layer.weights += layer.delta * input_to_use.T * learning_rate

    def train(self, X, y, learning_rate, max_epochs):
        """
        Trains the neural network using backpropagation.
        :param X: The input values.
        :param y: The target values.
        :param float learning_rate: The learning rate (between 0 and 1).
        :param int max_epochs: The maximum number of epochs (cycles).
        :return: The list of calculated MSE errors.
        """

        mses = []

        for i in range(max_epochs):
            for j in range(len(X)):
                self.backpropagation(X[j], y[j], learning_rate)
            if i % 10 == 0:
                mse = np.mean(np.square(y - nn.feed_forward(X)))
                mses.append(mse)
                print('Epoch: #%s, MSE: %f' % (i, float(mse)))

        return mses

    @staticmethod
    def accuracy(y_pred, y_true):
        """
        Calculates the accuracy between the predicted labels and true labels.
        :param y_pred: The predicted labels.
        :param y_true: The true labels.
        :return: The calculated accuracy.
        """

        return (y_pred == y_true).mean()


def my_func(x1, x2):
    return [math.sin(2*x1+x2)]

n = 40
np.random.seed(4)
x1_low, x1_up = -3, 3
x2_low, x2_up = -1, 3
x1s = np.random.uniform(x1_low, x1_up, size=n)
x2s = np.random.uniform(x2_low, x2_up, size=n)

Xs = []
for _x1 in x1s:
    for _x2 in x2s:
        Xs.append([_x1, _x2])

Zs = [my_func(_x1, _x2) for _x1, _x2 in Xs]

# Define test data
x1_pred = np.random.uniform(x1_low, x1_up, size=n)
x2_pred = np.random.uniform(x2_low, x2_up, size=n)
Xs_pred = []
for _x1, _x2 in zip(x1_pred, x2_pred):
    Xs_pred.append([_x1, _x2])
actual_ys = [my_func(_x1, _x2) for _x1, _x2 in Xs_pred]

# Train and test neural network
alpha = 0.001
nn = NeuralNetwork()
nn.add_layer(Layer(2, 5, 'tanh'))
nn.add_layer(Layer(5, 1, 'tanh'))
errors = nn.train(Xs, Zs, alpha, 300)
print('Accuracy: %.2f%%' % (nn.accuracy(nn.predict(Xs_pred), actual_ys) * 100))
$\endgroup$
1
$\begingroup$

Maybe your network is not complex enough to approximate your function. Try adding more layers or increasing the number of units in each layer (or both). And don't forget to add more training points if you increase the model size to avoid overfitting.
During training, check that both your train and test errors decrease.
I also advise you to generate your training points the same way you generated your test points (with zip). You training data will be more random and will better cover the input space.
You can do a function to generate N random points that you can call for training points and for test points.

def generate_points(n):
    x1 = np.random.uniform(x1_low, x1_up, size=n) 
    x2 = np.random.uniform(x2_low, x2_up, size=n) 
    return list(zip(x1, x2))
$\endgroup$
1
  • $\begingroup$ I have to use 2-5-1 network and can't extend it $\endgroup$
    – Naan
    Jun 9, 2019 at 21:13
1
$\begingroup$

Neural networks have a tons of hyperparameters you can tune to improve the result :

  • Test other activation functions for the hidden layer : relu or sigmoid for example
  • Add some training examples
  • Tune the learning rate : sometimes the learning rate is too high for a good convergence
  • Change the optimization strategy : SGD, Adam, genetic algorithm ...
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.