Backpropagation with a different sized training set?

Question

I'm trying to create a NN whose input is a (length m) array of 3d vectors $$\vec{x}_i = [x_{i,1},x_{i,2},x_{i,3}], \hspace{5mm}i=1:m $$

and whose output is a similarly sized array:

$$\vec{h}_{\theta,i} = [h_{\theta,i1},h_{\theta,i2},h_{\theta,i3}], \hspace{5mm}i=1:m $$

BUT, my only training data is not 3d vectors but rather the magnitude/norm of such vectors (with no knowledge of the vector components ($\lambda's$) themselves):

$$y_i= ||[\lambda_{i,1},\lambda_{i,2},\lambda_{i,3}]||, \hspace{5mm}i=1:m $$

So, my concept is to use the cost function:

$$ J = \frac{1}{2m}\sum (||\vec{h}_{\theta,i}|| - ||y_i||)^2 $$

Note, the difference between this and the more usual quadratic cost function ($ J = \frac{1}{2m}\sum (\vec{h}_{\theta,i} - \vec{y}_i)^2 $) which I would use if $y_i$ was the same shape as the output.

In the typical cost function case I've mentioned above the backpropagation algorithm always starts with computing the output layer error (which based on the simple derivative of that cost function is just $\delta^L = a^L - y_i $). From there you follow to the next layer error which depends on $\delta^L$.

My problem is what should my output layer error be? I've tried just using the derivative of $J$ with respect to $a^L$ using my new cost function (I won't write out that derivative here because it's a bit hairy), but I can't seem to simply slot that in as my output error and continue on with normal back propagation with any kind of sensible results?

If you can't suggest how I should approach my BP algorithm, then perhaps it's because you think my cost function is a non-starter? As a note: I have sort of proven the concept to myself using a linear model rather than an NN and training it with this dataset and linear derivatives rather than BP. The accuracy was low though given the nonlinearity of the data so I would like to use an NN approach.

Answer 1

Your cost function implies that you want the network to predict the magnitude of the output vector based on the magnitude of the input vector. Is that what you want to do? If not, e.g. if you want to accurately predict the output vector components, you need to believe that these are predictable based only on the magnitude of the inputs. If you don’t think this is possible, then you need to find new data. If you do, then a normal error function should be fine (e.g. MSE) and you can use normal backdrop as implemented in any NN library. If you’re getting poor performance, it’s probably because the norm of the input vector does not carry enough information to make a good prediction.

Answer 2

In modern libraries, such as tensorflow or pytorch, you can easily build custom loss functions and / or custom layers.

For example, the last layer could be a frozen layer which computes the square length of its 3d input. But there is no guarantee that the inputs of this artificial layer will converge to what you expect even if the final output (the length) is learned correctly. You will likely need to look for other methods or additional information to recover the output vectors and not just their lengths.