# Why does my model learn with Ragged Tensors but not Dense Tensors?

I have a string of letters that follow a "grammar." I also have boolean labels on my training set of whether the string follows "the grammar" or not. Basically, my model is trying to learn determine if a string of letters follows the rules. It's a fairly simple problem (I got it out of a textbook).

I am generating my dataset like this:

def generate_dataset(size):
good_strings = [string_to_ids(generate_string(embedded_reber_grammar))
for _ in range(size // 2)]
for _ in range(size - size // 2)]
X = tf.ragged.constant(all_strings, ragged_rank=1)

# X = X.to_tensor(default_value=0)

y = np.array([[1.] for _ in range(len(good_strings))] +
return X, y


Notice the line X = X.to_tensor(default_value=0). If this line is commented out, my model learns just fine. However, if it is not commented out, it fails to learn and the validation set performs the same as chance (50-50).

Here is my actual model:

np.random.seed(42)
tf.random.set_seed(42)

embedding_size = 5

model = keras.models.Sequential([
keras.layers.InputLayer(input_shape=[None], dtype=tf.int32, ragged=True),
keras.layers.Embedding(input_dim=len(POSSIBLE_CHARS) + 1, output_dim=embedding_size),
keras.layers.GRU(30),
keras.layers.Dense(1, activation="sigmoid")
])
optimizer = keras.optimizers.SGD(lr=0.02, momentum = 0.95, nesterov=True)
model.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(X_train, y_train, epochs=5, validation_data=(X_valid, y_valid))


I am using 0 as the default value for the dense tensors. The strings_to_ids doesn't use 0 for any of the values but instead starts at 1. Also, when I switch to using a Dense tensor I change ragged=True to False. I have no idea why using a dense tensor causes the model to fail, as I've used dense tensors before in similar exercises.

For additional details, see the solution from the book (exercise 8) or my own colab notebook. As a disclaimer: I am studying ML for fun, not for work or school.