1
$\begingroup$

Thank you all, I am playing around with a deterministic function in order to understand machine learning as in the tutorial blog at https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/#comment-503950 .

The program I am using a deterministic function y = f(x) where f(x) = x**2. I get a beautiful plot with the (x, predicted f(x)) and (x, f(x)). x and f(x) are scaled using MinMaxScaler. It follows that the predicted f(x) from the model is scaled.

So I want to rescale predicted f(x) to its normal. Here is the cut-down code.

Summary: plots beautifully when the value is scaled, BUT when the data is re-scaled, I don't understand why the re-scaled values of yhat = predicted f(x) are the same.

from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense
from numpy import asarray
from matplotlib import pyplot

# define data
x = asarray([i for i in range(1000)]); # This is x UNSCALED
y = asarray([a**2 for a in x]); # This is f(x) = x**2 UNSCALED

# reshape into rows and cols
x = x.reshape((len(x), 1));  # This is x UNSCALED
y = y.reshape((len(y), 1));  # This is f(x) unscaled

# scale data
x_s = MinMaxScaler()
x = x_s.fit_transform(x)     # This is x SCALED
y_s = MinMaxScaler()
y = y_s.fit_transform(y)     # This is f(x) SCALED - come back to it.

# fit a model
model = Sequential()
model.add(Dense(300, input_dim=1, activation='relu'));
model.add(Dense(1))
#compile a model
model.compile(loss='mse', optimizer='adam')

#fit a model
model.fit(x, y, epochs=277, batch_size=200, verbose=0)
mse = model.evaluate(x, y, verbose=0)
print("the value of the mse")
print(mse)

# predict
yhat = model.predict(x); #This is 'scaled' predicted value
# plot real vs predicted - the plots look beautiful
pyplot.plot(x,y,label='y')
pyplot.plot(x,yhat,label='yhat')
pyplot.legend()
print("the graph is printed on another window")
pyplot.show()

print("Printing the output of the scaled values of yhat, f(x) and x")
print("printing the first 10")
for i in range(10):
    print(yhat[i],y[i],x[i])
print("printing the 10th to 20th")
for i in range(10):
    print(yhat[i+10],y[i+10],x[i+10])

print("Printing the output of the unscaled values")

y_predicted = y_s.inverse_transform(yhat); # NOTE HERE using original UNSCALED y = f(x)'s y_s

y_expected = y_s.inverse_transform(y)

x_original = x_s.inverse_transform(x)
#print(y_predicted[0:10,].tolist(), x_original[0:10,].tolist())
print("Printing the first 10, predicted, expected, and x")
for i in range(10):
    print(y_predicted[i], y_expected[i], x_original[i])


print("let's try some other arbitrary section, say 10:20")
#print(y_predicted[9:21,].tolist(),y_predicted[9:21,].tolist(), x_original[9:21,].tolist())
print("printing 10th to 20th, predicted, expected, and x")
for i in range(10):
    print(y_predicted[i+10],y_expected[i+10], x_original[i+10])

But I don't know why the rescaled yhat to the original scale produces this result

Printing the output of the unscaled/re-scaled values
Printing the first 10, predicted, expected, and x
[1171.0186] [0.] [0.]
[1171.0186] [1.] [1.]
[1171.0186] [4.] [2.]
[1171.0186] [9.] [3.]
[1171.0186] [16.] [4.]
[1171.0186] [25.] [5.]
[1171.0186] [36.] [6.]
[1171.0186] [49.] [7.]
[1171.0186] [64.] [8.]
[1171.0186] [81.] [9.]
let's try some other arbitrary section, say 10:20
printing 10th to 20th, predicted, expected, and x
[1171.0186] [100.] [10.]
[1171.0186] [121.] [11.]
[1171.0186] [144.] [12.]
[1171.0186] [169.] [13.]
[1171.0186] [196.] [14.]
[1171.0186] [225.] [15.]
[1171.0186] [256.] [16.]
[1171.0186] [289.] [17.]
[1171.0186] [324.] [18.]
[1171.0186] [361.] [19.]

I would have expected the left-most column, which is the rescaled yhat = predicted f(x) to approximate the 2nd column, BUT ALL THE VALUES IN THE LEFT-MOST COLUMN ARE THE SAME.

I want to clarify that when I print the scaled values, the values of yhat(scaled) = predicted are all the same. So something is happening at the scaled value of yhat such that they're all the same. printing the first few scaled yhat, f(x) and x. Where they are displayed in groups of 3. Cannot use and in comments. [0.00117336] [0.] [0.], [0.00117336] [1.002003e-06] [0.001001], [0.00117336] [4.00801202e-06] [0.002002], [0.00117336] [9.01802704e-06] [0.003003],


I played around again with the experiment to determine the accuracy of deep learning being able to predict a deterministic function without using the formula y = f(x) = x**2.

I have an improved result, but need to work out why the estimated values, say the first five estimate predicted values seem out of place.

Yet the plots of (x,y) and (x,yhat) seemed to look quite close.

Here is the code to replicate the problem. I left out the import statements.

x = x.reshape((len(x),1))
y = y.reshape((len(y),1))
x_s = MinMaxScaler()
y_s = MinMaxScaler()
x_scaled = x_s.fit_transform(x)
y_scaled = y_s.fit_transform(y)
model = Sequential()
model.add(Dense(100,input_dim=1,activation='relu'))
model.add(Dense(1))

model.compile(loss='mse',optimizer='adam')
model.fit(x_scaled,y_scaled, epochs=100, batch_size=10,verbose=0)

mse = model.evaluate(x_scaled, y_scaled,verbose=0)
mse
1.0475558547113905e-05

yhat = model.predict(x_scaled)
yhat_original = y_s.inverse_transform(yhat)

#First five of yhat_original (yhat rescaled)
yhat_original[:5].T
array([[11.835742, 11.835742, 11.835742, 11.835742, 11.835742]]
#compared to first original 5 elements of y = 0,1,4,9,16

#Last five of yhat_original (yhat rescaled)
yhat_original[-5:].T
array([[8985.839, 9154.454, 9323.067, 9491.684, 9660.3  ]
#compared to last original 5 elements of y = 9025, 9216, 9409, 9604, 9801

#Now determine the RMS of the predicted and original values
cum_sum = 0
for i in range(len(yhat_original)):
    cum_sum+= (y[i]-yhat_original[i])**2/len(yhat_original)
mse = sqrt(cum_sum)
mse
array([31.72189417])
pyplot.plot(x,y,label='y')
pyplot.plot(x,yhat_original,label='estimated')
pyplot.legend()

Result: results were more accurate at the higher end but remained inaccurate at the lower end.

Any advice please, Thank you, Anthony of Sydney

$\endgroup$
0
$\begingroup$

Your results are fine. Try using the same plot function you used above for the entire dataset values:

pyplot.plot(x_original,y_expected,label='y')
pyplot.plot(x_original,y_predicted,label='yhat')
pyplot.legend()
print("the graph is printed on another window")
pyplot.show()

You will see that it looks good. The problem with the first few values are probably because both x and y are very close to 0, makes the estimation more difficult.

Edit: You also seem to have no shuffle anywhere. That also might be contributing to the problem.

$\endgroup$
  • $\begingroup$ Thank you for your reply. I did some further experimentation and added layers and varied the number of neurons between each layer. In essence, I looked at the plot of (x,f(x)) and (x,yhat) and found them to be very close. In addition when experimenting with layers and number of neurons in each layer, I looked at the lower levels of yhat and found them to be very very close to 0, 1, 4, 9, 16 and very little error. Still need to fine tune, and to examine the fit of the model using k-folds shuffling. $\endgroup$ – Anthony from Sydney Nov 14 '19 at 1:32
  • $\begingroup$ Even before k-fold validation, just a simple shuffling of your dataset once before training will probably improve your results. But you seem to use your entire dataset both for test and for training, which generally is not a good idea. $\endgroup$ – serali Nov 14 '19 at 7:27
  • $\begingroup$ Thank you for your post comment and thanks to the moderator for editing and deleting. $\endgroup$ – Anthony from Sydney Nov 15 '19 at 5:08
  • 1
    $\begingroup$ Dear 'serali' thank you again. I did the reshuffling of the x and f(x) and made a model on that basis. The result was a better fit. When it came to plotting the data, I plotted dots instead of a lineplot in order to avoid a zig-zag pattern on the plot. In addition I sorted/unshuffled the data back to produce a list of x, f(x) and yhat. Thank you again, Anthony of Sydney $\endgroup$ – Anthony from Sydney Nov 20 '19 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.