Confusion with tensorflow's Sequential Dense Layers

Question

I'm working on a regression probem using Tensorflow, and have created two models with slight differences in their first Dense layer.

The Models

# Create some regression data
X_regression = tf.range(0, 1000, 5)
y_regression = tf.range(100, 1100, 5) # Y = X+ 100

# Split regression data into training and test sets
X_reg_train = X_regression[:150]
X_reg_test = X_regression[150:]
y_reg_train = y_regression[:150]
y_reg_test = y_regression[150:]

Model 1

# Setup random seed
tf.random.set_seed(42)


model_1_reg = tf.keras.Sequential([
    tf.keras.layers.Dense(100),
    tf.keras.layers.Dense(10),
    tf.keras.layers.Dense(1)
])


model_1_reg.compile(loss=tf.keras.losses.mae,
                    optimizer=tf.keras.optimizers.Adam(),
                    metrics=['mae'])

model_1_reg.fit(tf.expand_dims(X_reg_train, axis=-1), y_reg_train, epochs=100)

Model 2

# Setup random seed
tf.random.set_seed(42)


model_2_reg = tf.keras.Sequential([
    tf.keras.layers.Dense(100, input_shape=(None, 1)),
    tf.keras.layers.Dense(10),
    tf.keras.layers.Dense(1)
])


model_2_reg.compile(loss=tf.keras.losses.mae,
                    optimizer=tf.keras.optimizers.Adam(),
                    metrics=['mae'])

model_2_reg.fit(tf.expand_dims(X_reg_train, axis=-1), y_reg_train, epochs=100)

I'm confused about whether I should add the input_shape or not. Model 1's input shape becomes (None, 1) and Model 2's input becomes (None, None, 1).

Both of them run, but perform differently.

Model 2 makes sense since we're inputting an array, but if I think about it, does that mean I only have a single node in the input layer? Since I'm giving it a whole ndarray instead of the instances it self. Model 1 makes sense too since I want to give each number into it.

So, which one makes sense more? Or what case should I use each model for? Also, for model 2's fit why does doing

tf.expand_dims(X_reg_train, axis=-1)

for the X of

model_2_reg.fit(tf.expand_dims(X_reg_train, axis=-1), y_reg_train, epochs=100

work? I thought we're suppose to put it in as a batch or like an array of the data so it should be inside an ndarray?

Answer 1

Keras expects the inputs $X$ to be batched, i.e., of shape $B\times D$ in your case (for a feed-forward NN), where $B$ is the batch size (by default is 32, which can be assigned in model.fit(..., batch_size=xyz)) and $D$ is the dimensionality of your data or, if you want, the number of features.

In your case, you have one feature and also both X_regression and y_regression are not batched, i.e., they have shape $(N,)$ instead of $(N,1)$. Therefore, by calling tf.expand_dims(x, axis=-1) or x[:, tf.newaxis] or even tf.reshape(x, shape=(-1, 1)), you can correctly reshape them.

At this point, you specify the correct input dimensionality in the Sequential model, by setting input_shape=(1,) in the first dense layer. Note that Dense layers can also work on rank-3 tensors (i.e., of shape $B\times D\times K$) so if you write input_shape=(None, 1) you're adding another dimension which will be inferred at the first forward pass, i.e., model(x). Basically, the batch dimension is implicit and you don't want to specify it.