Using keras LSTM implementation with sparse data

Question

I am trying to build an Autoencoder with LSTMs. My input data has been one hot encoded, which results in around 13.000 features.

My problem at this point is, that I get the following error message:

...
File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 926, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 229, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 208, in constant
value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/home/user/.local/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 447, in make_tensor_proto
"Cannot create a tensor proto whose content is larger than 2GB.")

ValueError: Cannot create a tensor proto whose content is larger than 2GB.

How can I reduce the size of the tensor without secraficing the deepth of the model respectively without rewriting the code in tensor myself?

I am using tesnorflow as my backend. My model looks as following:

# current values for timestamps and features: 1 and 12113
def build_model(timestamps, features):
    model = Sequential()
    layers = {'hidden1': 64, 'hidden2': 16, 'output': features}

    model.add(LSTM(
            batch_input_shape=(batch_size, timestamps, features), #batch_size is set to 500
            units=layers['hidden1'],
            return_sequences=True,
            stateful=True))

    model.add(LSTM(
            units=layers['hidden2'],
            return_sequences=True,
            stateful=True))

    model.add(LSTM(
            units=layers['output'],
            return_sequences=False,
            stateful=True))

    model.compile(loss="binary_crossentropy",
                    optimizer="rmsprop", # is recommended vor RNNs)
    return model