# How do I build a permutation invariance neural network in keras?

My question is about the structure of the network required to solve my problem with fewer data.

I have a sensor device that simply reports the color of the thing it's seeing in front of it. One sensor reports me 4 numbers: Red, Green, Blue, and Alpha. The intensity of the color changes depends on the distance and the thing it's seeing. I have 6 such sensors attached on each side of a small cube. The cube can be moved and rotated using the hand.

I want to predict the position of the cube in space in real-time.

My problem:

Input: 6 identical sensors, each giving 4 numbers. Total=6*4=24 numbers.

Output: 3 numbers, X, Y, Z, (Position of the cube)

I have my data ready with label XYZ.

Now, train a simple Multi-layer perceptron which takes 24 numbers and output 3 numbers. This is working quite fine but it needs hell lots of data in a cubic meter space to predict accurately.

The problem is about the rotation. I need to rotate and cover 360 degrees for every location for it to predict well.

But I know for the fact that each sensor is identical so I want to share the weight of each sensor. I know that when you rotate the cube 90 degrees, it should not affect the output position at all. So this should mean the order of the sensors is not important. This means I should somehow use add or average to merge my sensors' layer. If I use concatenate it will preserve the order which make the output position changes.

The way I'm doing it is that I feed 4 numbers into a sensor model that are shared among all sensors, get the encoding, add them up, then connect it to Dense layers. Following is the model prototype:

from keras.layers import Input, Dense
from keras.models import Model, Sequential

sensor1 = Input(shape=(4,))
sensor2 = Input(shape=(4,))
sensor3 = Input(shape=(4,))
sensor4 = Input(shape=(4,))
sensor5 = Input(shape=(4,))
sensor6 = Input(shape=(4,))
sensor_model = Sequential([
Dense(64, activation='relu'),
Dense(64, activation='relu'),
])
sensor1_encoding = sensor_model(sensor1)
sensor2_encoding = sensor_model(sensor2)
sensor3_encoding = sensor_model(sensor3)
sensor4_encoding = sensor_model(sensor4)
sensor5_encoding = sensor_model(sensor5)
sensor6_encoding = sensor_model(sensor6)
sensor_encoding = average([
sensor1_encoding,
sensor2_encoding,
sensor3_encoding,
sensor4_encoding,
sensor5_encoding,
sensor6_encoding,
])
h = sensor_encoding
h = Dense(128, activation='relu')(h)
h = Dense(128, activation='relu')(h)
h = Dense(3, activation='linear')
model = Model(inputs=[sensor1, sensor2, sensor3, sensor4, sensor5, sensor6], outputs=[h])


Right now, when I change the average function to concatenate, the model loss is lower on both training and validation set which contradicts with my intuition. What is wrong with my thinking? What do you think? How do I adjust this model in order for it to predict the same position if I rotate 90 degrees while also not suffering from 45 degrees rotation. And also make it in such a way that it does not remove the useful relationship between the inputs.

• You may want to use PCA and send its first n-outputs to the network. Commented Dec 18, 2018 at 8:30
• rotation invariance and permutation invariance are two VERY different things. Commented Dec 18, 2018 at 8:34
• @Media I don't like PCA as it removes useful data and it assumes linear relationships. Commented Dec 18, 2018 at 9:53
• @MartinThoma Thanks. I can see that it's quite different in the sense the rotation invariance have more constraint that it requires the 2 sides of the cube to be connected but permutation does not mean the 2 sides of the cube will be connected. Commented Dec 18, 2018 at 9:54
• But my problem can be considered permutation invariance too if you group 2 sensors from the opposite sides as one unit. Call that twin sensor model if you want. So totally I will have 3 such models that can permute in 6 ways. Suppose I have 3 twin sensors, each giving 8 numbers, this is exactly the same as permutation invariance, no? Commented Dec 18, 2018 at 9:55

Concatenate was fine.

By concatenating, you give to your network access to all information it needs to predict the output. The order does not matter.

By averaging, you crunch your data into something smaller : something that represent your data in a less informative way. That's why your loose accuracy : you are giving to your network less information to learn from.

What you need to do is get more data. Your initial approach is fine, you just need enough data for your network to learn that when the data represent a 90 degree-rotation, the output should be the same.

• This does not solve my issue about too much training data. I want the network to learn from one rotation that it's the same as the many other rotations. Commented Dec 18, 2018 at 9:57

I have found the solution. And I've written the code for it, for anyone interested please check: https://github.com/offchan42/superkeras/blob/master/permutational_layer.py

It's an implementation from a paper. It's about pairing all 2 inputs and use the model that takes in 2 inputs. The model are shared and the output of them are summarized using average. This preserve the permutational invariance in every layer while also making the input sees each other all the time.