# learning rate in reinforcement learning

Does anyone know how to get the learning rate from participant data?

I'm computing all the expected values for all trials (=200)

$$V_t(S) = V_{t-1}(S)+ \alpha \cdot \text{error }_t$$

$$(\text{error }_t = R_t − V_{t-1}(S))$$

and then computing alpha at each trial using

$$\alpha= (V_t(S) - V_{t-1}(S) )/ \text{error }_t$$

but it seems not the right way!

Suggestions?

• Can you explain why it doesn't seem like the right way? If you provide clear information on what's wrong, its easier for people to help you.
– Ana
Jun 25 '15 at 20:51
• So I initialised all the values in the first iteration.. computed the V(S)t and then I used all the V(S)t already computed to computed alpha trial-by-trial.. first, I really don't know if it is the right way of proceeding ..second I've got a alpha vector of all 1 and NaN values...
– Ire
Jun 26 '15 at 9:29

If you still have the question I hope my answer helps you. From what I understand you want to fit a RL model to participant's data who did a decision making task. The whole process of fitting RL models to data is described in great detail in Trial by Trial analysis using RL by Nathaniel Daw.

Briefly, what you observe (your data) is just actions and rewards at every time step. If the participant is using a RL model you should expect to update the values of his actions according to:

$Q_t(a_t)=Q_t(a_{t-1})+\alpha\cdot (r_t-Q_t(a_{t-1}))$

and action selection with the Boltzmann function:

$p(a_{t}|s_{t})=\frac{e^{\beta Q(s_{t},a_{t})}}{\sum_{a'}e^{\beta Q(s_{t},a')}}$

We prefer to use a stochastic policy as it simulates better human choices and it is straightforward to create a likelihood function. The likelihood function would be the product of all the probabilities $p(a_t|s_t)$. The model parameters are $[{\alpha,\beta}]$ and these can be inferred by standard Maximum Likelihood techniques.

Finally, our hypothesis (=model) can be summarized as: A subject generates a sequence of (observed) actions. To generate these actions we assumed that uses a RL model with some parameters. We want to fit the model to the data to infer these parameters and test the suitability of the model and the cognitive/psychological links of the values of the parameters.