In Q-learning, why does Q index on both state and action?

Question

In Q-learning, Q is an array of expected rewards for (state, action) combinations.

It seems to me the same result could be achieved while slightly simplifying the algorithm, if instead of associating reward with taking a given action in a state, we just associate it with the state. Then a reward would come to be associated with taking action A in state S, by virtue of the fact that this leads to state T, which has the reward. (This would correspond to the intuitive idea that an action is taken not for its own sake, but in order to steer the world into a desirable state.)

Is there a reason this is not done? All the references I have found so far, simply take it for granted that Q maps (state, action) pairs to expected rewards, without commenting on other possibilities.

Answer 1

What you described sounds just like the usual TD(0) learning of state value functions. If you only have state values (including the next state value) which by definition they must follow a specific bootstrapped (non-optimal) behavior policy $\pi_b$, then you don't have direct criterion to improve your policy in the absence of a model under Q-learning's assumption. In fact all kinds of RL algos ultimately try to find an optimal policy and follow the generalized policy iteration (GPI) idea which inherently utilizes action value function Q indexed on both state and action.

Mathematically if you want to solve the Bellman optimality equation (foundation of Q-learning) which is only expressed directly in terms of optimal state value function and actions allowed at that state you'll still end up with a greedy optimal policy solution in terms of the same Q function indexed on both state and action.