8
$\begingroup$
  • I randomly generate millions groups of triplet $\lbrace x_0, x_1, x_2 \rbrace$ within range $(0,1)$, then calculate the corresponding coefficients of the polynomial $(x-x_0)(x-x_1)(x-x_2)$, which result in triplet groups normalized in a form of $\lbrace { {x_0+x_1+x_2 \over 3} , {\sqrt{x_0x_1+x_1x_2+x_0x_2 \over 3}} , {\sqrt[3]{x_0x_1x_2}}} \rbrace$;

  • After that, I feed the coefficient triplets in to a 5-layered neural network $\lbrace 3,4,5,4,3 \rbrace$, in which all the activation function is set to sigmoid and the learning rate is set to 0.1;

  • However, I only get a very poor cross validation, around 20%.

How can I fix this?

BackGround

My original problem is a dynamic inverse problem. In that problem, I have hundreds of thousands of observations $O$, from these observations, I need to recover several hundred parameters $P$. The simulation process from $P$ to $O$ is very easy and cheap to calculate, but the inversion from $O$ to $P$ is highly nonlinear and nearly impossible. My idea is to train a neural network getting $O$ as inputs and $P$ as outputs. To check the feasibility of this idea, I employ a 3-ordered polynomial equation to do the validation.

update half a year later

With more nodes per layer, I have successfully trained a neural network. The topology is set to $\lbrace 3, 64, 64, 64 \rbrace$. And the most important trick is, sorting the generated triplet $\lbrace x_0, x_1, x_2 \rbrace$, ensuring $x_0 <= x_1 <= x_2$ always holds.

$\endgroup$
  • $\begingroup$ I would create a neural network to estimate the number of real roots then something like Newton's method to estimate them. $\endgroup$ – Emre Feb 9 '17 at 18:42
  • $\begingroup$ @Emre In the numeric experiment above, both the training and the validation sets, I always have three real roots in range (0,1). So I do not need to estimate the number of real roots in this case. $\endgroup$ – Feng Wang Feb 9 '17 at 19:49
  • $\begingroup$ I don't get the role of the polynomial by now. Could you please expand on this? Can't you collect many pairs (P, O) and take a subset of those for validation? $\endgroup$ – Martin Thoma Feb 9 '17 at 20:58
  • $\begingroup$ @MartinThoma I want to test my idea with a much simpler inversion problem. I mean, the coefficients triplet as observation $O$, and the roots triplet as parameter $P$ to recover, where the process from $\{x_0, x_1, x_2\}$ to polynomial coefficients is straightforward, but the inversion is much difficult. If I can successfully train such a neural network taking coefficients as inputs and roots as outputs, I would like to migrate this idea to my dynamic inverse problem in the next step. $\endgroup$ – Feng Wang Feb 9 '17 at 21:11
  • $\begingroup$ @MartinThoma I can always generate as many as possible pairs of $\{P, O\}$ to train and to validate the performance. $\endgroup$ – Feng Wang Feb 9 '17 at 21:18
6
$\begingroup$

You're trying to fit a very complicated function. There is no reason to expect that neural networks will be very good at this. Neural networks aren't magic pixie dust. They can do some things well, but don't expect a silver bullet.

You're trying to get the neural network to learn to compute some complicated function. We do know that given sufficiently nodes and sufficiently many layers, you can get an arbitrarily good approximation to this function, but there's no a priori reason to expect it should be possible with the particular (small) number of nodes and layers you chose. In fact, two layers suffices, if you have sufficiently many nodes -- but we have no way to compute how many nodes are needed.

In particular, the function you are trying to compute amounts to computing $x_0,x_1,x_2$ from $a,b,c,d$, where

$$\begin{align*} x_k &= - {1 \over 3a} \left(b + \eta^k C + {\Delta_0 \over \eta^k C}\right)\\ \Delta_0 &= b^2 - 3ac\\ C &= \sqrt[3]{\Delta_1 \pm \sqrt{\Delta_1^2 - 4\Delta_0^3} \over 2}\\ \Delta_1 &= 2b^3 - 9abc + 27a^2d\\ \eta &= -{1 \over 2} + {1 \over 2} \sqrt{3} i \end{align*}$$

(This comes from the general formula for solving a cubic equation.)

That's an extremely messy function, and thus it might be challenging to approximate using a neural network of the architecture you present. You can always try increasing the number of nodes and/or number of layers, but there's no a priori theory to tell you what the right number is.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.