0
$\begingroup$

I was trying to implement a no hidden layer model for classification with the constraint that all the weights have the absolute value $1$, As follows

import tensorflow as tf
from tensorflow.keras.constraints import max_norm
from tensorflow.keras.layers import Dense

model = tf.keras.models.Sequential()
#model.add(kernel_constraint=max_norm(1))
model.add(Dense(4, kernel_constraint=max_norm(1)))
model.add(Dense(2,activation='sigmoid',use_bias=False))
#model.add(tf.keras.layers.Dense(2, activation='sigmoid',use_bias=False))
model.compile(loss = 'mse',metrics='accuracy')
model.fit(X_train,y_train,batch_size= 100, epochs=30) 

Here the input has four features, output has two feature with 0 bias. The training phase converges with very good accuracy. So there should be $4*2 = 8$ parameters in the model. But

model.get_weights().shapemodel.get_weights().shape  

has returned a list which is

[array([[ 0.33709845, -0.55206317, -0.53412324, -0.15946062],
        [ 0.03901154,  0.11299153,  0.0941186 , -0.31381565],
        [ 0.69983363, -0.06287932, -0.14378403,  0.22027704],
        [-0.6285503 ,  0.13892443,  0.26558703, -0.42299727]],
       dtype=float32),
 array([ 0.30418202,  0.29648915, -0.31335464,  0.3056269 ], dtype=float32),
 array([[ 1.1184788 , -0.24037583],
        [ 0.7782192 , -0.11152704],
        [-0.01809523,  0.38015074],
        [ 0.3689365 , -0.7091095 ]], dtype=float32)]

The parameter size does not look a $4*2$ output. Seems like $4*4*2$ is the shape I got. Please help me to know where is the problem in my code and how to make the shape $4*2$ keeping the weight constraint. Also the third array has one weight $> 1$, violate the constraints. Though the model is trained well.

$\endgroup$

1 Answer 1

1
$\begingroup$

The model you have made actually has a hidden layer, i.e. the 4 neuron layer is actually a hidden layer

The number of parameters is affected by the input size you put in

eg:

model = tf.keras.models.Sequential()
model.add(Dense(4, kernel_constraint=max_norm(1)))
model.add(Dense(2,activation='sigmoid',use_bias=False))
model.compile(loss = 'mse',metrics='accuracy')
model.fit([[1,2,3,4]],[[1,0]])  ## the input shape is (4,)
model.summary()

Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_5 (Dense)              (None, 4)                 20        
_________________________________________________________________
dense_6 (Dense)              (None, 2)                 8         
=================================================================
Total params: 28
Trainable params: 28
Non-trainable params: 0
_________________________________________________________________

If you want to create a model with input layer of size 4 you should instead

model = tf.keras.models.Sequential()
model.add(Input(shape=(4,)))
model.add(Dense(2,activation='sigmoid',use_bias=False))
model.compile(loss = 'mse',metrics='accuracy')
model.summary()

which gives the desired model:

Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_7 (Dense)              (None, 2)                 8         
=================================================================
Total params: 8
Trainable params: 8
Non-trainable params: 0
_________________________________________________________________

The number of trainable parameters i.e. the weights is equal to the total params in model.summary()

$\endgroup$
2
  • $\begingroup$ Can you please set the parameter constrained to have maximum norm value 1, $\endgroup$ Jun 27, 2021 at 18:49
  • $\begingroup$ does model.add(Dense(2,activation='sigmoid',use_bias=False, kernel_constraint=max_norm(1))) as the last layer solve the problem? $\endgroup$ Jun 27, 2021 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.