What are features for state-action pairs in RL?

Question

I read this answer: What are features in the context of reinforcement learning?

But it only describes features for the state only in the context of cartpole, ie. Cart Position, Cart Velocity, Pole Angle, Pole Velocity At Tip

On slide 18 here: http://www.cs.cmu.edu/~rsalakhu/10703/Lecture_VFA.pdf

It states:

But does not give examples. I started reading from p. 198 in Sutton's book for Value Function Approximation but also did not see examples for "features of state-action pairs" .

My best guess is for example in Cartpole-V1 (discrete action space) would be to add one more number to the tuple describing the state-action pair, ie. (Cart Position, Cart Velocity, Pole Angle, Pole Velocity At Tip, push_right) . In the case of Cartpole I guess each state action pair could be described with a feature vector of length 3 where the final input for the tuple is either "push_left", "do_nothing", "push_right".

Would the immediate reward from taking one of the actions also be included in the tuples that form the state-action feature vector?

Answer 1

In the cartpole example, a state-action feature could be

$$\begin{bmatrix} \text{Cart Position}\\ \text{Cart Velocity}\\ \text{Pole Angle}\\ \text{Pole Tip Velocity}\\ \text{Action} \end{bmatrix}$$

where Action is either left, right, or do nothing. The reward is not part of the feature vector because reward does not describe the state of the agent; it is not an input. It is a (possibly stochastic) signal received from the environment that the agent is trying to predict/control with the use of feature vectors.

Answer 2

A feature vector is a vector that is containing basis functions. These basis functions are combining states and actions. We can use a feature vector to approximate our action-value function $q(\boldsymbol{s},\boldsymbol{a})$. If for example $\boldsymbol{\Phi}(s,a)=[\phi_0, \phi_1(\boldsymbol{s},\boldsymbol{a}), \ldots,\phi_m(\boldsymbol{s},\boldsymbol{a})]^T.$

Then the action-value function can be approximated by

$$q^{\pi}(\boldsymbol{s},\boldsymbol{a}) = \boldsymbol{w}^T\boldsymbol{\Phi}(s,a),$$

in which $\boldsymbol{w}=[w_0, w_1,\ldots,w_m]^T$ are coefficients that we need to determine.

These basis functions might be something very complicated. For the mountain cart problem, we will not even be able to visualize them because we would need a $\text{dim}\,\mathcal{S}\,\text{dim}\,\mathcal{A} + 1$ dimensional coordinate system.

But we can try to come up with an example which might allow us to visualize the basis functions. Imagine a line with $5$ nodes which correspond to the states $\mathcal{S}=\{s_1, s_2, s_3, s_4, s_{5,\text{terminal}}\}$. The last state is the terminal state. Additionally, we assume that we can go to the left $a=-1$ or to the right $a=+1$. Hence, the action space is $\mathcal{A}=\{-1,+1\}$. We assume that the environment is deterministic, which means that if we want to go right/left we will go in this direction. Additionally, we assume that we have a deterministic policy $\pi(s)=1$, hence we will always go right. We assume that every step costs us $-1$. If we are in the first state and we go right we will get a reward of $-2$ and still be at the first state. In the terminal state, the reward is $0$.

For this problem, we can start to determine all values of the action-value function $q^{\pi}(s,a)$. The following table

\begin{array}{ccc} \hline s& a & q^{\pi}_{s,a} \\ \hline 1 & 1 & -4\\ 1 & -1 & -6\\ 2 & 1 & -3\\ 2 & -1 & -5\\ 3 & 1 & -2\\ 3 & -1 & -4\\ 4 & 1 & -1\\ 4 & -1 & -3\\ \hline \end{array}

The exact solution $q^{\pi}(s,a)$ is given by

$$q^{\pi}(s,a)=\sum_{\tilde{s},\tilde{a}}q^{\pi}_{\tilde{s},\tilde{a}}\delta(s-\tilde{s})\delta(a-\tilde{a}).$$

In which $\delta(x)$ is $1$ if $x=0$ and $0$ for all other $x$. You can see that the exact feature vector would consist of these basis functions $\phi_{\tilde{s},\tilde{a}}(s,a)=\delta(s-\tilde{s})\delta(a-\tilde{a})$.

But we could also approximate action-value function by

$$\hat{q}(s,a)=-6+s+a,$$

which would yield $\phi_0=-6$, $\phi_1(s,a)=s$ and $\phi_2(s,a)=a$. This approximation would perfectly approximate the exact action-value function at the specified values of $s$ and $a$. But it is using different basis functions (which are the features).