I wrote a blog post where I calculated the Taylor Series of an autoregressive function. It is not strictly the Taylor Series, but some variant (I guess). I'm mostly concerned about whether the derivatives look okay. I noticed I made a mistake and fixed the issue. It seemed simple enough,but after finding an error, I started to doubt myself.

$$f(t+1) = w_{t+1} \cdot f(t) $$

$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$

$$y^{*}_{t+1} = w_{t+1} f(t) + \dfrac{d}{df(t)}w_{t+1}f(t) + \dfrac{d}{dw_{t+1}}w_{t+1}f(t)$$

$$y'_{t+1} = w_{t+1} f(t) + w_{t+1} + f(t)$$

The details can be found in the blog post:

EDIT 7/6/20:

The AR form:

$$y^{*}_{t+1}=c+\sum _{{I=0}}^{L}w _{t+1-i}y_{{t-i}}+\varepsilon _{t}$$

f(t) is a recursive dense layer, y is the predicted output, and w are the weights, and L are the number of lag components. For the simple case where the next value only depends on the previous value, I got the following result.

$$f(t+1) = w_{t+1} \cdot f(t) $$

$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$

$$y^{*}_{t+1} = w_{t+1} f(t) + \dfrac{d}{df(t)}w_{t+1}f(t)$$

$$y'_{t+1} = w_{t+1} f(t) + w_{t+1}$$

EDIT 7/7/20:

The function f(t) represents y(t) with an error term. The error term might have some random process, but I'm going to assume that the errors are independent.

$$f(t+1) = w_{t+1} \cdot y(t) + \epsilon_t$$

EDIT 7/9/20:

Changed the dimensionality of w_t+1 to w_t.

$$f(t+1) = w_{t} \cdot f(t) $$

$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$

$$y^{*}_{t+1} = w_{t} f(t) + \dfrac{d}{df(t)}w_{t}f(t)$$

$$y'_{t+1} = w_{t} f(t) + w_{t}$$

  • 1
    $\begingroup$ I am not sure I understand the post, nor why the chain rule (or perhaps you mean the product rule) for derivatives would result in the 3rd equation... $\endgroup$ – Nikos M. Jul 5 '20 at 16:46
  • $\begingroup$ Oh you are right, it is actually the partial derivative + the product rule. $\endgroup$ – FreedomToWin Jul 5 '20 at 21:02
  • $\begingroup$ The purpose was to linearize the recursive formula using the Taylor Series approximation. I couldn't find any resources on how to do this so I took a stab at it. $\endgroup$ – FreedomToWin Jul 6 '20 at 1:30
  • 1
    $\begingroup$ Where is the actual AR (auto-regressive) or ARMA (auto-regressive-moving average) model? Where are these equations. it seems you start directly from trying to find the adaptive rule for the weights. For example what is $f(t)$? I presume $y_t$ is the output and $w_t$ is the weight. But where is the actual model? Please update your question with the model equation first, then try to find the adaptive rule for weights $\endgroup$ – Nikos M. Jul 6 '20 at 9:06
  • 1
    $\begingroup$ OK but I dont make much sense of this definition of $f(t)$. Why does it multiply $y_t$ with $w_{t+1}$ (doesnt make much sense, dimensional analysis, for example, fails)? You already have an error term $\epsilon_t$ in the definition of the AR model. Why is this function needed, what purpose does it fit? $\endgroup$ – Nikos M. Jul 9 '20 at 6:33

I guess the differentiation is wrong.

$y^{*}_{t+1} = w_{t+1}f(t) + \frac{df(t+1)}{d(t+1)}$

$y^{*}_{t+1} = w_{t+1}f(t) + \frac{d(w_{t+1}f(t))}{dt}*\frac{dt}{d(t+1)}$

$y^{*}_{t+1} = w_{t+1}f(t) + w_{t+1}\frac{df(t)}{dt} + f(t)\frac{dw_{t+1}}{dt}$

  • $\begingroup$ Thanks for the answer! Hmm, t is not actually a parameter though. It would need to in terms of w_t. $\endgroup$ – FreedomToWin Jul 5 '20 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.