Self answering after reading this article. Actually, the paragraph is saying that $218$ new features are obtained from -
- Memory cells or Memory units of 2nd LSTM layer: It will give a vector of $100$ cell states (cell state corresponding to each memory cell). Note that the dimension of the vector is equal to the number of hidden memory cells.
- Last output of 2nd LSTM layer: It will give a list of $100$ cell outputs (cell output of each memory cell/unit).
- Final output: Final output would eventually comprise of an $18$ dimensional vector. Dimensions are - $2$ for (a), $6$ for (b), $4$ for (c), $3$ for (d), $2$ for (e), $1$ for (f).
From the same article, it can be achieved in keras using return_state=True
parameter.
from keras.models import Model
from keras.layers import Input
from keras.layers import LSTM
from numpy import array
# define model
inputs1 = Input(shape=(3, 1))
lstm1, state_h, state_c = LSTM(2, return_sequences=True, return_state=True)(inputs1)
model = Model(inputs=inputs1, outputs=[lstm1, state_h, state_c])
# define input data
data = array([0.1, 0.2, 0.3]).reshape((1,3,1))
# make and show prediction
print(model.predict(data))
This ouputs,
[array([[[-0.00559822, -0.0127107 ],
[-0.01547669, -0.03272599],
[-0.02800457, -0.0555565 ]]], dtype=float32), array([[-0.02800457, -0.0555565 ]], dtype=float32), array([[-0.06466588, -0.12567174]], dtype=float32)]