Problem in the parameter size in constraint parameter no hidden netwrok

Question

I was trying to implement a no hidden layer model for classification with the constraint that all the weights have the absolute value $1$, As follows

import tensorflow as tf
from tensorflow.keras.constraints import max_norm
from tensorflow.keras.layers import Dense

model = tf.keras.models.Sequential()
#model.add(kernel_constraint=max_norm(1))
model.add(Dense(4, kernel_constraint=max_norm(1)))
model.add(Dense(2,activation='sigmoid',use_bias=False))
#model.add(tf.keras.layers.Dense(2, activation='sigmoid',use_bias=False))
model.compile(loss = 'mse',metrics='accuracy')
model.fit(X_train,y_train,batch_size= 100, epochs=30) 

Here the input has four features, output has two feature with 0 bias. The training phase converges with very good accuracy. So there should be $4*2 = 8$ parameters in the model. But

model.get_weights().shapemodel.get_weights().shape  

has returned a list which is

[array([[ 0.33709845, -0.55206317, -0.53412324, -0.15946062],
        [ 0.03901154,  0.11299153,  0.0941186 , -0.31381565],
        [ 0.69983363, -0.06287932, -0.14378403,  0.22027704],
        [-0.6285503 ,  0.13892443,  0.26558703, -0.42299727]],
       dtype=float32),
 array([ 0.30418202,  0.29648915, -0.31335464,  0.3056269 ], dtype=float32),
 array([[ 1.1184788 , -0.24037583],
        [ 0.7782192 , -0.11152704],
        [-0.01809523,  0.38015074],
        [ 0.3689365 , -0.7091095 ]], dtype=float32)]

The parameter size does not look a $4*2$ output. Seems like $4*4*2$ is the shape I got. Please help me to know where is the problem in my code and how to make the shape $4*2$ keeping the weight constraint. Also the third array has one weight $> 1$, violate the constraints. Though the model is trained well.

Answer 1

The model you have made actually has a hidden layer, i.e. the 4 neuron layer is actually a hidden layer

The number of parameters is affected by the input size you put in

eg:

model = tf.keras.models.Sequential()
model.add(Dense(4, kernel_constraint=max_norm(1)))
model.add(Dense(2,activation='sigmoid',use_bias=False))
model.compile(loss = 'mse',metrics='accuracy')
model.fit([[1,2,3,4]],[[1,0]])  ## the input shape is (4,)
model.summary()

Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_5 (Dense)              (None, 4)                 20        
_________________________________________________________________
dense_6 (Dense)              (None, 2)                 8         
=================================================================
Total params: 28
Trainable params: 28
Non-trainable params: 0
_________________________________________________________________

If you want to create a model with input layer of size 4 you should instead

model = tf.keras.models.Sequential()
model.add(Input(shape=(4,)))
model.add(Dense(2,activation='sigmoid',use_bias=False))
model.compile(loss = 'mse',metrics='accuracy')
model.summary()

which gives the desired model:

Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_7 (Dense)              (None, 2)                 8         
=================================================================
Total params: 8
Trainable params: 8
Non-trainable params: 0
_________________________________________________________________

The number of trainable parameters i.e. the weights is equal to the total params in model.summary()