3
$\begingroup$

In machine learning, "feature" is a synonym for explanatory variables. I know what a feature is. However, in the specific case of RL, it's not clear to me what features are. What are "features" in the specific case of RL? What are examples of features in RL? Which RL algorithms require the specification of features? Why do we need "features" in RL?

$\endgroup$

1 Answer 1

1
$\begingroup$

What are "features" in the specific case of RL?

In RL, the supervised learning component is used to learn a function approximation for either a value function or a policy. This is usually a function of the state, and sometimes of the state and action. The features are therefore explanatory variables relating to the state (or state and action) and need to be sufficient to explain the value or optimal action of that state.

In RL literature, there is often no difference between terms "state representation" or "features". Although you may need to process the state representation using feature engineering, into suitable features for whatever supervised learning you are using. With neural networks, this may involve normalising values that might otherwise cause problems.

What are examples of features in RL?

In OpenAI's gym, CartPole-v1 has the following features describing state:

  • Cart Position
  • Cart Velocity
  • Pole Angle
  • Pole Velocity At Tip

The positions and speeds are measured in arbitrary units in this case, but you might consider them to be in SI units - metres along the track, metres per second, radians, metres per second.

Typically in RL, you will want features that:

  • Describe the state (and possibly the action, depending on implementation)

  • Are feasible to collect by observation of the environment

  • Together with the action choice, can adequately explain expected rewards and next state. In RL terms, they should in aggregate possess the Markov property

Which RL algorithms require the specification of features?

None of them*. However, without descriptive features of state, you are limited to enumerating all possible states, and using tabular methods. Working with tabular methods constrains you in practice by the time it takes to fully explore all the states, and the memory space required to represent all the individual estimates of value.

If you are looking into solving your problem using DQN, A3C or other "deep" RL method that uses neural networks, then you need to be thinking in terms of a state representation that is composed of features, and to treat those features as if they were inputs to the same neural networks used in supervised learning.

Why do we need "features" in RL?

To work with complex environments with large state spaces that cannot be solved without some form of approximation and (hopefully) generalisation.


* This is slightly more complex with policy gradient methods, as they do require that you use a function approximator, such as a neural network. However, it is possible to make the "features" in that case a one-hot coded vector of the state enumeration, which is the same representation as tabular value-based methods. There is little reason to do that in practice, but it is possible in principle.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy