Approximating multi-variable function with neural network in python

I'm trying to use 2-5-1 neural network to approximate function $$x_1 \in[-3,3], x_2\in[-1,3], f(x_1,x_2)=\sin(2x_1+x_2)$$

I used code from Implementing a flexible neural network with backpropagation from scratch, to avoid using any complex libraries and tried teaching my network to approximate with following data:

# Define dataset
n = 40
np.random.seed(4)
x1_low, x1_up = -3, 3
x2_low, x2_up = -1, 3
x1s = np.random.uniform(x1_low, x1_up, size=n)
x2s = np.random.uniform(x2_low, x2_up, size=n)

Xs = []
for _x1 in x1s:
for _x2 in x2s:
Xs.append([_x1, _x2])

Zs = [my_func(_x1, _x2) for _x1, _x2 in Xs]

# Define test data
x1_pred = np.random.uniform(x1_low, x1_up, size=n)
x2_pred = np.random.uniform(x2_low, x2_up, size=n)
Xs_pred = []
for _x1, _x2 in zip(x1_pred, x2_pred):
Xs_pred.append([_x1, _x2])
actual_ys = [my_func(_x1, _x2) for _x1, _x2 in Xs_pred]

# Train and test neural network
for ee in range(0, 4):
for e in range(1, 4):
alpha = e / 10 ** ee
nn = NeuralNetwork()
errors = nn.train(Xs, Zs, alpha, 300)
print('Accuracy: %.2f%%' % (nn.accuracy(nn.predict(Xs_pred), actual_ys) * 100))
# Plot changes in mse
plt.plot(errors)
plt.ylim([0, 1])
plt.title(str('Changes in MSE - alpha ' + str(alpha)))
plt.xlabel('Epoch (every 10th)')
plt.ylabel('MSE')
plt.show()

But I can't seem to have MSE lower than $$0.4$$. What can I do here to make it more accurate?

• what have you tried so far?
– oW_
May 31, 2019 at 21:22
• I change max epochs, learning rate, and size data to learn from. I'm wondering if I have some error in my attached code, that prevents me from finding solution. All my approximations equal zero...
– Naan
May 31, 2019 at 23:09

One problem that I see is that notice that sine is a function that takes value from $$-1$$ to $$1$$ but sigmoid function takes value from $$0$$ to $$1$$.

Hence you are being penalized whenever the sine value takes negative value.

You might like to try to change your last layer to a tanh layer or alternatively, rather than predicting sine directly, predict $$\frac{\sin(2x_1+x_2)+1}{2}$$ first.

I manage to achieve MSE of $$0.228686$$ using the tanh modification. Of course, you can still try to tune other parameters and try other stuff to improve the model.

import numpy as np
import matplotlib.pyplot as plt
import math

class Layer:
"""
Represents a layer (hidden or output) in our neural network.
"""

def __init__(self, n_input, n_neurons, activation=None, weights=None, bias=None):
"""
:param int n_input: The input size (coming from the input layer or a previous hidden layer)
:param int n_neurons: The number of neurons in this layer.
:param str activation: The activation function to use (if any).
:param weights: The layer's weights.
:param bias: The layer's bias.
"""

self.weights = weights if weights is not None else np.random.rand(n_input, n_neurons)
self.activation = activation
self.bias = bias if bias is not None else np.random.rand(n_neurons)
self.last_activation = None
self.error = None
self.delta = None

def activate(self, x):
"""
Calculates the dot product of this layer.
:param x: The input.
:return: The result.
"""

r = np.dot(x, self.weights) + self.bias
self.last_activation = self._apply_activation(r)
return self.last_activation

def _apply_activation(self, r):
"""
Applies the chosen activation function (if any).
:param r: The normal value.
:return: The "activated" value.
"""

# In case no activation function was chosen
if self.activation is None:
return r

# tanh
if self.activation == 'tanh':
return np.tanh(r)

# sigmoid
if self.activation == 'sigmoid':
return 1 / (1 + np.exp(-r))

return r

def apply_activation_derivative(self, r):
"""
Applies the derivative of the activation function (if any).
:param r: The normal value.
:return: The "derived" value.
"""

# We use 'r' directly here because its already activated, the only values that
# are used in this function are the last activations that were saved.

if self.activation is None:
return r

if self.activation == 'tanh':
return 1 - r ** 2

if self.activation == 'sigmoid':
return r * (1 - r)

return r

class NeuralNetwork:
"""
Represents a neural network.
"""

def __init__(self):
self._layers = []

"""
Adds a layer to the neural network.
:param Layer layer: The layer to add.
"""

self._layers.append(layer)

def feed_forward(self, X):
"""
Feed forward the input through the layers.
:param X: The input values.
:return: The result.
"""

for layer in self._layers:
X = layer.activate(X)

return X

def predict(self, X):
"""
Predicts a class (or classes).
:param X: The input values.
:return: The predictions.
"""

ff = self.feed_forward(X)

# One row
if ff.ndim == 1:
return np.argmax(ff)

# Multiple rows
return np.argmax(ff, axis=1)

def backpropagation(self, X, y, learning_rate):
"""
Performs the backward propagation algorithm and updates the layers weights.
:param X: The input values.
:param y: The target values.
:param float learning_rate: The learning rate (between 0 and 1).
"""

# Feed forward for the output
output = self.feed_forward(X)

# Loop over the layers backward
for i in reversed(range(len(self._layers))):
layer = self._layers[i]

# If this is the output layer
if layer == self._layers[-1]:
layer.error = y - output
# The output = layer.last_activation in this case
layer.delta = layer.error * layer.apply_activation_derivative(output)
else:
next_layer = self._layers[i + 1]
layer.error = np.dot(next_layer.weights, next_layer.delta)
layer.delta = layer.error * layer.apply_activation_derivative(layer.last_activation)

# Update the weights
for i in range(len(self._layers)):
layer = self._layers[i]
# The input is either the previous layers output or X itself (for the first hidden layer)
input_to_use = np.atleast_2d(X if i == 0 else self._layers[i - 1].last_activation)
layer.weights += layer.delta * input_to_use.T * learning_rate

def train(self, X, y, learning_rate, max_epochs):
"""
Trains the neural network using backpropagation.
:param X: The input values.
:param y: The target values.
:param float learning_rate: The learning rate (between 0 and 1).
:param int max_epochs: The maximum number of epochs (cycles).
:return: The list of calculated MSE errors.
"""

mses = []

for i in range(max_epochs):
for j in range(len(X)):
self.backpropagation(X[j], y[j], learning_rate)
if i % 10 == 0:
mse = np.mean(np.square(y - nn.feed_forward(X)))
mses.append(mse)
print('Epoch: #%s, MSE: %f' % (i, float(mse)))

return mses

@staticmethod
def accuracy(y_pred, y_true):
"""
Calculates the accuracy between the predicted labels and true labels.
:param y_pred: The predicted labels.
:param y_true: The true labels.
:return: The calculated accuracy.
"""

return (y_pred == y_true).mean()

def my_func(x1, x2):
return [math.sin(2*x1+x2)]

n = 40
np.random.seed(4)
x1_low, x1_up = -3, 3
x2_low, x2_up = -1, 3
x1s = np.random.uniform(x1_low, x1_up, size=n)
x2s = np.random.uniform(x2_low, x2_up, size=n)

Xs = []
for _x1 in x1s:
for _x2 in x2s:
Xs.append([_x1, _x2])

Zs = [my_func(_x1, _x2) for _x1, _x2 in Xs]

# Define test data
x1_pred = np.random.uniform(x1_low, x1_up, size=n)
x2_pred = np.random.uniform(x2_low, x2_up, size=n)
Xs_pred = []
for _x1, _x2 in zip(x1_pred, x2_pred):
Xs_pred.append([_x1, _x2])
actual_ys = [my_func(_x1, _x2) for _x1, _x2 in Xs_pred]

# Train and test neural network
alpha = 0.001
nn = NeuralNetwork()
errors = nn.train(Xs, Zs, alpha, 300)
print('Accuracy: %.2f%%' % (nn.accuracy(nn.predict(Xs_pred), actual_ys) * 100))

Maybe your network is not complex enough to approximate your function. Try adding more layers or increasing the number of units in each layer (or both). And don't forget to add more training points if you increase the model size to avoid overfitting.
During training, check that both your train and test errors decrease.
I also advise you to generate your training points the same way you generated your test points (with zip). You training data will be more random and will better cover the input space.
You can do a function to generate N random points that you can call for training points and for test points.

def generate_points(n):
x1 = np.random.uniform(x1_low, x1_up, size=n)
x2 = np.random.uniform(x2_low, x2_up, size=n)
return list(zip(x1, x2))
• I have to use 2-5-1 network and can't extend it
– Naan
Jun 9, 2019 at 21:13

Neural networks have a tons of hyperparameters you can tune to improve the result :

• Test other activation functions for the hidden layer : relu or sigmoid for example