Yes, actually exactly that. I just thought about it since my policies are not stationary. At each time step I have a different policy distribution. In common applications the policies are stationary; you use the same policy network over and over for each time step. But anyway, your answers were really helpful for me, thank you very much!

Would the following work then, based on the discussion so far; like in my previous comment: First optimizing $E[r_2]$ with respect to $\pi_2(a_2|s_2)$, using a vanilla policy gradient algorithm, without using actor critics or value functions and then optimizing $E[r_1 + V_2(s_2)]$ with respect to $\pi_1(a_1|s_1)$ by using the $V_2$ which we learn from the first optimization procedure. So there would be two separate neural networks for the two policies $\pi_1$ and $\pi_2$ in the end.

Oh, so I see that Q learning assumptions are about an infinte horizon case. Would policy gradients help me then? I think it is possible to maximize $V_2(s)$ wrt $\pi_2$ and then using them, frozen, for maximizing $V_1(s)$ wrt $\pi_1, as proposed here: rll.berkeley.edu/deeprlcoursesp17/docs/lec1.pdf

Many thanks for the detailed answer. Is the main problem in my case that the Bellman equation does not hold for me right? Is it only valid for problems with infinite time steps? I didn't exactly get what violates the Bellman equation in my case.

Thanks for the answer. Yes, basically we assume that we are in a Markovian system where all information we currently have depends just on the last step, we have a MDP. Sometimes I see some authors dropping time indices altogether when defining the Bellman Equality, so I wondered given common assumptions, is the value function stationary so we can work without the time index.

Thanks for the answer. So, we can safely assume that treating them as constant inputs is the usual practice, right?