0
$\begingroup$

I am dealing with a regression problem, for which I wanted to try to use a recurrent neural network. The general setting is that I have to predict a continuous quantity starting from the evolution, in time, of some other features. I have 23 features, and, for each of them, I have their evolution in time for seven timestamps.

Since the time interval between two consecutive sampling instants is not constant, I would like to implement a T-LSTM (a reference to https://dl.acm.org/doi/10.1145/3097983.3097997, there is an explanatory scheme of the network cell in the fourth page). I tried to implement the recurrent cell for the single-time sample and then wrap it into tf.keras.layers.RNN() class. The cell would be the combination of a tf.keras.layers.LSTMCell() preceded by a time-decay block, since the two are not interfering with each other. For now, I would like to use only the last output state of the layer to predict the final value (so using somehow the recurrent layer as an encoder).

The problem I am having regards the input shape of the model.

I would like to pass the entire [(batch, n_timestamps, features), (batch, n_timestamps)] - the first one contains the data, the second the elapsed time between two consecutive timestamps -, which in my case is [(None, 7, 23), (None, 7)]. However, I can only pass the single timestep to the network, and the input that is accepted without errors is [(None, 23), (None, 1)]. In this way, I can not specify the length of the sequence, even if it is known at the beginning. I understand that this is the input for a single cell, but I wonder if there is a way to define an a-priori shape for the timestamps.

Also, I am not sure how the network works. By using a random test tensor as input, shape=[(20,7,23), (20,7)], it accesses the cell seven times, and this is expected; but each time it accesses it with dimension shape=[(20,23), (20,1)] (I have a print inside the cell's self.call(...) method that I have not reported in the code below). Is this behavior expected?

Finally, the last question is: what should I do to specify the input size of the network in terms of timestamps? So far the output shape of the RNN layer is (None, None, n_units), but in this way, I can not use a Dense layer, after flattening, across all the output sequences.

Any help is truly appreciated!

Here is the code for the cell class:

class CustomLSTMCell(Layer):
  def __init__(self, units, **kwargs):
    super().__init__(**kwargs)
    self.units = units
    self.lstm_cell = LSTMCell(units, activation='tanh', recurrent_activation='sigmoid')
    self.state_size = [tf.TensorShape([units]), tf.TensorShape([units])]
    self.output_size = [tf.TensorShape([units])]

  def build(self, input_shapes):

    super().build(input_shapes)
    self.W_decay = self.add_weight(name='weight_decay', shape=(self.units,), 
                                   initializer='random_normal', trainable=True)
    self.b_decay = self.add_weight(name='bias_decay', shape=(self.units,),
                                   initializer='random_normal', trainable=True)


  def call(self, inputs, states):
    x_t, m_t = inputs
    h_tm1, c_tm1 = states

    # Custom decay computation
    C_s = tf.tanh(tf.multiply(c_tm1, self.W_decay) + self.b_decay)
    Chat_s = tf.math.divide(C_s, m_t)
    C_t = tf.math.subtract(c_tm1, C_s)
    c_tm1_new = tf.math.add(C_t, Chat_s)

    h_t, [h_t, c_t] = self.lstm_cell(x_t, states=[h_tm1, c_tm1_new])

    return h_t, [h_t, c_t]


    def get_config(self):
        return {"units": self.units}

The code for the model instantiation is:

n_units = 16
cells = CustomLSTMCell(units=n_units)
custom_lstm = RNN(cells, return_sequences=True)

# Inputs
input_1 = tf.keras.Input((None, 23))    # x_t: input (vector)
input_2 = tf.keras.Input((None, 1))     # m_t: additional term (scalar)

# custom layer
out = custom_lstm((input_1, input_2))


model = tf.keras.models.Model(inputs=[input_1, input_2], outputs=out)

# Compile and fit the model as usual
model.compile(optimizer='adam', loss='mse')
model.summary()

For the sake of completeness, I also report the model summary:

______________________________________________________________________________________
Layer (type)                Output Shape                 Param #   Connected to                  
======================================================================================
 input_1 (InputLayer)        [(None, None, 23)]           0         []                            
                                                                                                  
 input_2 (InputLayer)        [(None, None, 1)]            0         []                            
                                                                                                  
 rnn (RNN)                   (None, 16)                   2592      ['input_1[0][0]',             
                                                                     'input_2[0][0]']                 
                                                                                                  
======================================================================================
Total params: 2592 (10.12 KB)
Trainable params: 2592 (10.12 KB)
Non-trainable params: 0 (0.00 Byte)
______________________________________________________________________________________
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.