0
$\begingroup$

I'm working on a regression probem using Tensorflow, and have created two models with slight differences in their first Dense layer.

The Models

# Create some regression data
X_regression = tf.range(0, 1000, 5)
y_regression = tf.range(100, 1100, 5) # Y = X+ 100

# Split regression data into training and test sets
X_reg_train = X_regression[:150]
X_reg_test = X_regression[150:]
y_reg_train = y_regression[:150]
y_reg_test = y_regression[150:]

Model 1

# Setup random seed
tf.random.set_seed(42)


model_1_reg = tf.keras.Sequential([
    tf.keras.layers.Dense(100),
    tf.keras.layers.Dense(10),
    tf.keras.layers.Dense(1)
])


model_1_reg.compile(loss=tf.keras.losses.mae,
                    optimizer=tf.keras.optimizers.Adam(),
                    metrics=['mae'])

model_1_reg.fit(tf.expand_dims(X_reg_train, axis=-1), y_reg_train, epochs=100)

Model 2

# Setup random seed
tf.random.set_seed(42)


model_2_reg = tf.keras.Sequential([
    tf.keras.layers.Dense(100, input_shape=(None, 1)),
    tf.keras.layers.Dense(10),
    tf.keras.layers.Dense(1)
])


model_2_reg.compile(loss=tf.keras.losses.mae,
                    optimizer=tf.keras.optimizers.Adam(),
                    metrics=['mae'])

model_2_reg.fit(tf.expand_dims(X_reg_train, axis=-1), y_reg_train, epochs=100)

I'm confused about whether I should add the input_shape or not. Model 1's input shape becomes (None, 1) and Model 2's input becomes (None, None, 1).

Both of them run, but perform differently.

Model 2 makes sense since we're inputting an array, but if I think about it, does that mean I only have a single node in the input layer? Since I'm giving it a whole ndarray instead of the instances it self. Model 1 makes sense too since I want to give each number into it.

So, which one makes sense more? Or what case should I use each model for? Also, for model 2's fit why does doing

tf.expand_dims(X_reg_train, axis=-1)

for the X of

model_2_reg.fit(tf.expand_dims(X_reg_train, axis=-1), y_reg_train, epochs=100

work? I thought we're suppose to put it in as a batch or like an array of the data so it should be inside an ndarray?

$\endgroup$
2
  • 1
    $\begingroup$ Keras infers the input shape (see this question). What do you mean with "Both of them run, but perform differently"? Also, note that stacking multiple Dense layers is equivalent to a single Dense layer; you should include non-linearities (e.g. ReLU) in the intermediate layers. Check this answer for details on this issue. $\endgroup$
    – noe
    Nov 14 at 7:22
  • $\begingroup$ Ah that makes sense, okay. Should the input layer be (none, 1) or should it be (1,)? If I set (none, 1) it means that in the fit I'm only fitting in 1 instance of the data If i give an ndarray. However, if I set it as (1,), that means that the ndarray of integers are treated as a batch, so I'm giving it multiple instances. $\endgroup$ Nov 14 at 9:20

1 Answer 1

2
$\begingroup$

Keras expects the inputs $X$ to be batched, i.e., of shape $B\times D$ in your case (for a feed-forward NN), where $B$ is the batch size (by default is 32, which can be assigned in model.fit(..., batch_size=xyz)) and $D$ is the dimensionality of your data or, if you want, the number of features.

In your case, you have one feature and also both X_regression and y_regression are not batched, i.e., they have shape $(N,)$ instead of $(N,1)$. Therefore, by calling tf.expand_dims(x, axis=-1) or x[:, tf.newaxis] or even tf.reshape(x, shape=(-1, 1)), you can correctly reshape them.

At this point, you specify the correct input dimensionality in the Sequential model, by setting input_shape=(1,) in the first dense layer. Note that Dense layers can also work on rank-3 tensors (i.e., of shape $B\times D\times K$) so if you write input_shape=(None, 1) you're adding another dimension which will be inferred at the first forward pass, i.e., model(x). Basically, the batch dimension is implicit and you don't want to specify it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.