1
$\begingroup$

I want to choose an architecture that can deal with an input symmetry.

As input, I have a sequence of zeros and ones, like [1, 1, 1, 0, 1, 0] and at the output layer I have N neurons that outputs a categorical distribution like [0.3, 0.4, 0.3].

How can I force a NN to ouput the same distribution when I feed its reverse copy, i.e [1, 1, 1, 0, 1, 0]?

A simple way just to learn twice:

feed straight [1, 1, 1, 0, 1, 0] -> [0.3, 0.4, 0.3]
feed reverse  [0, 1, 0, 1, 1, 1] -> [0.3, 0.4, 0.3] 

Or maybe, there are more "elegant" ways? What type of architecture I should use or maybe I need to play with loss functuions?

$\endgroup$
0
1
$\begingroup$

The most general and robust way is to double the training set using the "mirror image" of each sample. Also it is sound approach from a statistical learning perspective.

There can be some approaches to symmetry where one hacks the NN architecture (eg by symmetrising the network) but these are usualy very fragile and do not generalise well, while the above approach is robust and lets the network adjust itself as it needs.

A similar but cheaper approach is to sort the samples before feeding them to the network. Eg for your example:

If you have this input [1, 1, 1, 0, 1, 0] then the sorted related mirror input (in lexicographic order) is the [0, 1, 0, 1, 1, 1] (reverse). So you train the NN only the sorted lexicographic inputs and before testing each sample you convert it to the sorted lexicographic related input (by reversing it) if it is not already in that form.

So effectively you normalise the samples (by reversing them if necessary) both before training and before testing. This is similar approach to doubling the train set, only it is cheaper (and in practice exactly accurate, while doubling the training set is only theoreticaly accurate, in practice it may have discrepancies).

$\endgroup$
2
  • $\begingroup$ I do not understand what is the puppose of sorting input samples, could you explain for a layman? $\endgroup$ – Kenenbek Arzymatov May 5 at 19:32
  • $\begingroup$ So you have one sample [x,y,z,w] you reverse it and it becomes [w,z,y,x] if it is lexicographicaly before [x,y,z,w], then you use that sample both to train and test, else you use the original sample as is. it is simply a normalisation process $\endgroup$ – Nikos M. May 5 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.