# Normalize the output of a dense layer with linear activation

I have the following architecture of my network:

def net_one(message):
weight1 = np.random.normal(loc=0.0, scale=0.01, size=[16, 16])
init1 = tf.constant_initializer(weight1)
out1 = tf.layers.dense(inputs=message, units=16, activation=tf.nn.relu, kernel_initializer=init1)
weight2 = np.random.normal(loc=0.0, scale=0.01, size=[16, 7])
init2 = tf.constant_initializer(weight2)
out2 = tf.layers.dense(inputs=out1, units=7, activation=None, kernel_initializer=init2)
return out2


Now as the output of the network is linear (None in tensorflow corresponds to a linear activation function), the output is unbounded. I need the square of the 2-norm of the output to be a constant, n (for energy constraint purposes). I do not want to use sigmoid or tanh as they hamper the performance. I tried the following:

code = net_one(input_bits)
code = code * tf.sqrt(n) / tf.linalg.norm(code)


I have two questions:

1. Does it achieve what I expect it to achieve?
2. Is there any better way (if this is indeed right) or any alternate way to achieve this?

Your way seems to be correct. I would suggest one other way that you might want to try (maybe it won't make your life better, but still): Since you want your data points to basically live on a sphere, you could train the angles in spherical coordinates. Fix the radius to $$\sqrt{n}$$ and this way your network has to learn one dimension fewer. I'm not sure how this will affect performance though.

One way to implement this in tensorflow could be

def spherical(input):
layer = []

for i in range(input.get_shape()[1]+1):
elem = 1
for j in range(i):
elem = elem*tf.sin(input[:,j])
if i < input.get_shape()[1]:
elem = elem*tf.cos(input[:,i])
layer.append(elem)

return tf.transpose(tf.convert_to_tensor(layer))


This will map any input (a tensor with shape=(ndatapoints, ndims)) to the unit sphere in ndims+1. By multiplication and shifting you could map it to any sphere you want. It is probably not the most elegant way, but it gets the job done. The mapping is not injective, if you would need it to be, you would have to make sure the elements of input stick to the respective intervals.

If you end up trying this, let me know how it went please ;)

• the idea looks promising, but can you please mention how to implement such a neural network? Oct 9 '19 at 14:21
• I edited a possible implementation into the answer. Oct 12 '19 at 18:07