3
$\begingroup$

I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m $$

and whose output is a similarly sized array:

$$\vec{h}_{\theta,i} = [h_{\theta,i1},h_{\theta,i2},h_{\theta,i3}], \hspace{5mm}i=1:m $$

BUT, my only training data is not 3d vectors but rather the magnitude/norm of such vectors (with no knowledge of the vector components ($\lambda's$) themselves):

$$y_i= ||[\lambda_{i,1},\lambda_{i,2},\lambda_{i,3}]||, \hspace{5mm}i=1:m $$

So, my concept is to use the cost function:

$$ J = \frac{1}{2m}\sum (||\vec{h}_{\theta,i}|| - ||y_i||)^2 $$

Note, the difference between this and the more usual quadratic cost function ($ J = \frac{1}{2m}\sum (\vec{h}_{\theta,i} - \vec{y}_i)^2 $) which I would use if $y_i$ was the same shape as the output.

In the typical cost function case I've mentioned above the backpropagation algorithm always starts with computing the output layer error (which based on the simple derivative of that cost function is just $\delta^L = a^L - y_i $). From there you follow to the next layer error which depends on $\delta^L$.

My problem is what should my output layer error be? I've tried just using the derivative of $J$ with respect to $a^L$ using my new cost function (I won't write out that derivative here because it's a bit hairy), but I can't seem to simply slot that in as my output error and continue on with normal back propagation with any kind of sensible results?

If you can't suggest how I should approach my BP algorithm, then perhaps it's because you think my cost function is a non-starter? As a note: I have sort of proven the concept to myself using a linear model rather than an NN and training it with this dataset and linear derivatives rather than BP. The accuracy was low though given the nonlinearity of the data so I would like to use an NN approach.

$\endgroup$
0
$\begingroup$

Your cost function implies that you want the network to predict the magnitude of the output vector based on the magnitude of the input vector. Is that what you want to do? If not, e.g. if you want to accurately predict the output vector components, you need to believe that these are predictable based only on the magnitude of the inputs. If you don’t think this is possible, then you need to find new data. If you do, then a normal error function should be fine (e.g. MSE) and you can use normal backdrop as implemented in any NN library. If you’re getting poor performance, it’s probably because the norm of the input vector does not carry enough information to make a good prediction.

$\endgroup$
2
  • $\begingroup$ Thanks for your response. To answer your first question, no my aim is not simply to predict the magnitude, it is to predict the three components of the vector, but using magnitude data, which I am confident is possible given my success training a linear model like I alluded to in the post (i.e. where I used $\vec{h}_{i} = A\vec{x}_i + b$ and trained for A and b rather than an NN training for $\theta$). Could you explain to me in more detail how this should be possible using MSE? The basic MSE function should imply $y$ is the same shape as $h$, but here I'm using 3d $h$ and 1d $y$? $\endgroup$
    – Spinach
    Apr 10 '20 at 20:29
  • $\begingroup$ The MSE is the mean squared error between what you predicted (h) and the correct value. i.e. If you predicted (1, 2, 3) and the real value was (1, 4, 2), the error would be calculated based on those two vectors, which are the same shape. My understanding is that for each observation in your training data you are trying to predict a 3-vector based on a scalar value. If so, this doesn’t require any special algorithms - normal backpropagation of error based on a well known cost function is fine. MSE may not be your best option - that was just an example! $\endgroup$ Apr 10 '20 at 23:17
0
$\begingroup$

In modern libraries, such as tensorflow or pytorch, you can easily build custom loss functions and / or custom layers.

For example, the last layer could be a frozen layer which computes the square length of its 3d input. But there is no guarantee that the inputs of this artificial layer will converge to what you expect even if the final output (the length) is learned correctly. You will likely need to look for other methods or additional information to recover the output vectors and not just their lengths.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.