4
$\begingroup$

Neural networks in many domains (audio, video, image text/NLP) can achieve great results. In particular in NLP using a mechanism named attention (transformer, BERT) have achieved astonishing results - without manual preprocessing of the data (text documents).

I am interested in applying neural networks to time-series. However, in this domain, it looks like most people apply manual feature engineering by either:

  • transposing the matrix of events to hold columns for each time observation and a row for each thing (device, patient, ...)
  • manually generating sliding windows and feeding snippets to an RNN/LSMTM.

Am I overlooking something? Why can't I find people using attention? Wouldn't this be much more convenient (automated)?

Improve this question
$\endgroup$
6
  • $\begingroup$ Those operations you apply are hardly manual feature engineering. Manual feature engineering usually refers to something more substantial for extracting human-designed features. $\endgroup$
    – D.W.
    Nov 28, 2020 at 22:11
  • $\begingroup$ But then the whole magic practically sits in finding the right data loader? $\endgroup$
    – Georg Heiler
    Nov 29, 2020 at 6:19
  • $\begingroup$ Sorry, I don't know what you are referring to. Perhaps ask a new question, where you can provide all the details? $\endgroup$
    – D.W.
    Nov 29, 2020 at 8:10
  • $\begingroup$ Let me clarify: my expectation/hope would be that similar to other domains (NLP) also for time-series neural networks simply just work when feeding the raw data. But From your comment and also various observations I learn that this is not the case. Rather transformations (feature engineering) are required. You are correct that it might be debatable what is manual, still I would love to have something which just works similar to cats & dogs classification for time-series without extensive preprocessing. $\endgroup$
    – Georg Heiler
    Nov 29, 2020 at 8:24
  • $\begingroup$ I think you have misinterpreted my comment. I never said that. Separately: transposing the matrix or generating sliding windows is a trivial step. It is a far cry from manual feature engineering; you could probably do it in one line, or a few lines of code. If that is all that is needed for a neural network to be effective, then I think it would be fair to say that neural networks just work. $\endgroup$
    – D.W.
    Nov 29, 2020 at 8:35

1 Answer 1

Reset to default
5
$\begingroup$

It is an interesting question. I would not completely agree with you though when you say that most time-series models dont use attention. However there is not as much documentation available on the web as there is for other applications.

LSTNet was one of the first papers that proposed using an LSTM + attention mechanism for multivariate forecasting time series. Temporal Pattern Attention for Multivariate Time Series Forecasting by Shun-Yao Shih et al. focused on applying attention specifically attuned for multivariate data.

Attend and Diagnose leverages self attention on medical time series data. This time series data is multivariate and contains information like a patient’s heart rate, SO2, blood pressure, etc.

A good link to further study this would be: https://towardsdatascience.com/attention-for-time-series-classification-and-forecasting-261723e0006d

Further quoting form the above paper: "self-attention and related architectures have led to improvements in several time series forecasting use cases, however, altogether they have not seen widespread adaptation. This likely revolves around several factors such as the memory bottleneck, difficulty encoding positional information, focus on pointwise values, and lack of research around handling multivariate sequences. Additionally, outside of NLP many researchers are not probably not familiar with self-attention and its potential."

I dont completely agree with the last statement, nonetheless I do agree that the benefits of attention have not yet captured the attention of researchers outside of NLP to the extent that it should have

Improve this answer
$\endgroup$
1
  • $\begingroup$ A follow-up writeup addressing a more specific question of using self-attention and parts of transformer encoder arch in a time-series domain which is also relevant to OPs question: datascience.stackexchange.com/questions/93377/… $\endgroup$
    – Allohvk
    Nov 2, 2021 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.