0
$\begingroup$

I am trying to perform a very simple experiment, predict the input number. The concept is same as an auto-encoder. But with just one layer, which can handle the task of encoding and decoding-

Also, wanted to observe how the network learns, with different training examples.

Initially, I took 10000 training samples, of integers, in range[1, 10000]. So, example if I pass 767676 as one of the test sample it should predict the output as 767676.0 This test passed very well. The activation function used here was 'softplus', which keeps the values in range [0, inf). the network -

train_d = numpy.array([i for i in range(10000)], dtype=np.int)

model = Sequential()
model.add(Dense(1, input_shape=(1, ), activation='softplus'))

model.fit([train_d], train_d, epochs=5000, batch_size=256, shuffle=True)

Now when I gave it 10000 training sample of decimal values in the range[-1, 1]. E.g. 0.3456 the expected output will be 0.3456 but rather, its giving me 0.5721. The network -

train_d = numpy.array[round(random.uniform(-1, 1), 4) for i in range(10000)], dtype=np.float)

model = Sequential()
model.add(Dense(1, input_shape=(1,), activation='softsign'))
model.fit([train_d], train_d, epochs=5000, batch_size=256, shuffle=True)
$\endgroup$
0

1 Answer 1

3
$\begingroup$

First of all, during the 10000 integer test, did you use all the integers from 0 to 9999 in training? If yes, then you have fully covered the whole input range. This means that while testing, you actually feed the network with data that are identical to the training data, therefore the accuracy is very high. What is the result of the network if you train it with 10000 randomly sampled integers in range of 10,000 and then test it with ALL integers in range of 10000? This test will reveal if your encoder generalizes well enough.

Also keep in mind that the smaller the number, the more difficult it is for the network to train due to vanishing gradient. Therefore, decimal numbers can saturate the learning process and lead to lower accuracy. Try changing the batch size (use smaller batches) and see if you decrease your output error.

$\endgroup$
1
  • 1
    $\begingroup$ My testing set consists of integers number greater than 10000 and less than 1000000, i have achieved accuracy of 98 %. The problem that I have solving is completely linear, what gets in comes out, so the transformation has to happen linearly, but I am using non-linear transformation function which is softsign for floating numbers in range [-1, 1]. I have to solve the problem by, using an identity loss, or by using more complex network architecture. In trying to use a complex architecutre gave me 97% accuracy. Also, you are correct, reducing the batch_size has helped. $\endgroup$ Jun 5, 2018 at 6:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.