1
$\begingroup$

Below is one example Attention-based Encoder-decoder network for multivariate time series forecasting task. I want to visualize the attention weights.

input_ = Input(shape=(TIME_STEPS,N))
x = attention_block(input_)
x = LSTM(512, return_sequences=True)(x)
x = LSTM(512)(x)
x = RepeatVector(n_future)(x)
x = LSTM(128, activation='relu', return_sequences=True)(x)
x = TimeDistributed(Dense(128, activation='relu'))(x)
x = Dense(1)(x)
model = Model(input_,x)
model.compile(loss="mean_squared_error",optimizer="adam",metrics=["acc"])
print(model.summary())

Here is the implementation of my attention block:

def attention_block(inputs):
    x=Permute((2,1))(inputs)
    x=Dense(TIME_STEPS,activation="softmax")(x)
    x=Permute((2,1),name="attention_prob")(x)
    x=multiply([inputs,x])
return x

I will highly appreciate if a fresh implementation of the attention model is provided.

$\endgroup$

1 Answer 1

1
$\begingroup$

A reference: https://github.com/zhaocq-nlp/Attention-Visualization/blob/master/exec/plot_heatmap.py. It plots attentions based on matplotlib

$\endgroup$
2
  • 1
    $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$
    – Ethan
    Jun 24, 2022 at 18:31
  • $\begingroup$ I need the attention values, i.e. the probability values, and if you have any example of time series forecasting, that helps. Thank you. $\endgroup$ Jul 28, 2022 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.