Typically the definition I see for the $Q$ and value functions is $$ Q^\pi(s_t, a_t) = \mathbb{E}_\tau\left[\sum_{t'=t}^T\gamma^{t'-t}r(s_{t'}, a_{t'})\ |\ s_t, a_t\right] \\ V^\pi(s_t) = \mathbb{E}_\tau\left[\sum_{t'=t}^T\gamma^{t'-t}r(s_{t'}, a_{t'})\ |\ s_t\right] $$ where the expectation is taken over trajectories. If $T = \infty$ (that is, in an infinite time horizon), $Q^\pi(s_t, a_t)$ and $V^\pi(s_t)$ do not depend on time. However, for finite time horizons, it seems like they are time dependent: even if $s_2 = s_3$ are the same state, we can have $V^\pi(s_2)\neq V^\pi(s_3)$.
However, I often see people talking about the value and Q-functions as if they aren't dependent on time, irrespective of the value of $T$. For example, when I was learning about policy evaluation, where we fit a function approximator to the value function, the training data we use is $\{(s_t^i, \hat{V}^\pi(s_t^i))\}$ (sampled from a bunch of roll-outs) - this makes it seem like the timestep $t$ is irrelevant: if $s_2^i = s_3^j$, we're expecting the neural network to map both $s_2^i$ and $s_3^j$ to two different values: $\hat{V}^\pi(s_2^i)$ and $\hat{V}^\pi(s_3^j)$. Obviously a function approximator could just fit to the "average" value, but something still seems awry in its formulation.
As other evidence, I often see people write $V^\pi(s)$ for the value function, even in a finite horizon setting, making no mention of the time parameter. In implementations of Q-learning I've seen, the $Q$ function is represented as a 2d-array, not a 3d-array (meaning it's only dependent on the state and the action, not on a timestep).
Can someone help me wrap my head around my misunderstanding? How is it that we can effectively ignore the time parameter? Is is that for large enough $T$, the problem is effectively infinite horizon (since $\gamma^{t'-t}$ becomes so small)?
