Questions tagged [reinforcement-learning]
Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.
621
questions
0
votes
0
answers
36
views
Examples of decision processes where action has no effect on the next state, but has on the reward?
Within my studies, I work on a recurrent reinforcement learning project and I struggle to find real-world problems with a property that is important to my solutions. I look for instances of problems ...
- 106
0
votes
0
answers
14
views
Reinforcement Learning: Formally, does an exponentially decaying epsilon satisfy GLIE?
I am aware that exponentially decaying exploration constants (epsilon) are used practically. Formally, though, do they satisfy the GLIE condition?
Specifically, relating this to the Borel-Cantelli ...
0
votes
0
answers
13
views
Should offline algorithms (SAC, DDGP, TD3) be used with suprocesses vectorized environements?
I read most of offline algorithms from Stable Baselines 3 should not be subprocessed (see here : https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html - "Which algorithm should I ...
- 1
0
votes
0
answers
18
views
What are typical hyperparameter ranges and significance for SB3's offline algorithm?
I'm trying to perform an hyperparameter tuning on a SAC algorithm (Stable Baselines 3) with optuna lib.
I guess hyperparameters have different significance and typical ranges depending of the selected ...
- 1
0
votes
0
answers
30
views
Machine learning can be broken down into supervised, unsupervised, and reinforcement learning. Is there anything else?
Is it even logically possible to have a type of machine learning that doesn't fall into those three paradigms?
Supervised: a dataset of inputs and outputs are fed to an algorithm which learns a ...
0
votes
1
answer
36
views
Reinforcement learning
I am working on a sentiment analysis project. I used BERT model for training but lack of data it gives huge overfitting. So after i moved LLM approach to do that. Using LLM finally i got good results....
0
votes
0
answers
17
views
Bootstrapping formula for TD learning in Reinforcement Learning
In Reinforcement Learning (Sutton & Barto, 2018), p.120, equations (6.3)-(6.4)
, to explain the idea of bootstrapping in Temporal-difference learning:
\begin{equation}
v_{\pi}(s) := E_{\pi}[G_t|...
0
votes
1
answer
54
views
Deep Q-Learning: How are network parameters updated, and why consider episodes in the first place?
I'm trying to wrap my head around the implementation of deep $Q$-learning, and why we even consider episodes in the first place. The usual set-up is that we initialize some starting state $s_0$, then ...
- 101
0
votes
0
answers
19
views
Is n-Step SARSA Off-Policy?
To keep things simple, let us consider the case of an episodic Finite MDP with n=2 and no discounts. Copying the equation for n-step SARSA from Reinforcement Learning 2nd edition, page 146:
The n-step ...
- 11
0
votes
0
answers
18
views
Building an application to group/label browser tabs
I'm trying to develop an application where users can click a button and all of the open tabs in their browser will be placed into tab groupings based on similarity of the tab. Microsoft Edge has a ...
- 1
0
votes
0
answers
17
views
SARSA with Updated Actions
I am trying to gain a better understanding of the Classic SARSA Algorithm. Let us assume Finite MDP which is simple enough so that we can use Tabular methods, so no function approximation. In the ...
- 11
0
votes
0
answers
19
views
Query modification for elastic search using machine learning
I have a problem statement that I'm struggling to formulate as a machine learning framework. There is a huge client database of documents - we're trying to come up with an efficient way of querying ...
- 167
1
vote
1
answer
141
views
Given historical states and action/reward data, is reinforcement learning a reasonable approach?
To summarize my problem:
I want to maximize my total reward over all timesteps
I have 3 discrete actions at each time step.
The state vector for each time step has 5 features. The features are ...
- 251
0
votes
1
answer
71
views
Workflow for reinforcement learning in Python
I'm new to reinforcement learning but (I believe) I have certain experience with machine learning. I know if you work on, say, a classification problem, your workflow will be (in a very high level) ...
0
votes
0
answers
188
views
Choosing the number of episodes and iterations when training a RL model
I know the parameters chosen for training a RL model depend heavily on the model itself as well as the problem. Nevertheless, I am trying to train a bunch of these agents on different environments, ...
0
votes
0
answers
22
views
Can somebody explain to me, how the Hyperparameters in this expamle work? I only uderstood the meaning for gamma
I am quite new to this topic, but I want to understand how Reinforcement Learning (RL) works in this example (https://gymnasium.farama.org/tutorials/training_agents/reinforce_invpend_gym_v26/). I have ...
- 1
0
votes
1
answer
38
views
Stick to R or switch to Python as a newby in Reinforcement Learning?
Background
I would consider myself to have a decent level in R and virtually 0 skills in any other programming language - something quite common in my field. Note also that my background is not in ...
- 3
0
votes
0
answers
85
views
Reinforcement learning with PPO - tips and tricks
I am trying to use PPO where the agent has to maneuver around an obstacle towards the target while respecting the spatial boundaries. While the agent learns to respect spatial boundaries it never ...
2
votes
0
answers
15
views
Consistency error in visualization of policy improvement in Sutton & Barto's book?
Sutton & Barto introduce in their foundational book on "Reinforcement Learning: An Introduction" in the context of Dynamic Programming algorithms for policy evaluation and improvement. ...
- 41
0
votes
0
answers
24
views
Pretrained model for RNN Encoder-Decoder?
Our team are implementing a paper called Cold-Start-Reinforcement-Learning-with-Softmax-Policy-Gradient.
Although the paper didn't mention. We want to use a pre-trained model, which is a RNN Encoder-...
- 25
0
votes
1
answer
42
views
Gradients Exploding with Custom Gradient and 2 layer MLP
I am training a 2 layer MLP on an off-policy learning-to-rank task, where the input is a list of documents against a query with a feature vector for each query-document pair, i.e. if there are M ...
- 3,775
0
votes
1
answer
25
views
Inverse reinforcement learning with trajectories only
Inverse reinforcement learning (IRL) is a task that can learn a reward from other agent behaviour. Most IRL paradigms assume that dynamics of environment is known, that is the transition probability ...
- 515
0
votes
0
answers
26
views
Is it true that a multi-armed-bandit selects an action based on the future reward?
During my machine learning lecture I saw the following statement on the slides: "Multi-armed-bandit selects an action based on the future reward"
Is that statement true?
In my opinion it is ...
0
votes
0
answers
18
views
Dueling DQN with varying number of actions
I have an RL problem, where the number of actions depends on the state. Furthermore, each action-value computation requires action information in the form of a high-dimensional, continuous vector in ...
0
votes
0
answers
24
views
State and next state is the same on openai gym atari
i am using OpenGym atari to trainning my Pacman agent, this is part of my code
...
- 1
0
votes
0
answers
32
views
find q-table for discrete action space
I am trying to use q-learning for a discrete observation space that is represented by:
buffer: list of 200 integer values in [0,10]
discard_counter: list of 200 integer values in [0, 4]
capacity: ...
- 1
0
votes
0
answers
16
views
Multiagent observations encoding - LSTM or Transformer?
In multi agent learning problems where agents can join or leave, to address the issue of varying observation space I am trying to build a latent representation either using LSTM or transformer which ...
0
votes
0
answers
14
views
Policy Gradient training log-derivative un-normalized vs normalized objective
I am implementing a policy gradient training objective for optimizing ranking metrics in a learning-to-rank setting. For a given query $q$, a set of documents $D_q$ (retrieved from a first-stage ...
- 3,775
0
votes
0
answers
23
views
SB3-PPO and Keras-PPO on CartPole env: performances can be improved and catastrophic forgetting limited?
I am trying to experiment different hyperparameters and neural network structures in order to stabilize the learning process of the CartPole environment and possibly reduce the occurrence and gravity ...
- 61
0
votes
0
answers
14
views
Llimit/masking the space of actions to the legal ones only: A -> A(S)
I have difficulties on coding a way to limit the available actions as a function of the current agent state S. I am trying to use https://keras.io/examples/rl/ppo_cartpole/ (that works quite fine) and ...
- 61
1
vote
0
answers
52
views
A working tf_agents.agents.PPOClipAgent example for reference?
I am really struggling on finding a working example for tf_agents.agents.PPOClipAgent, for instance for the simple CartPoleV1 environment. Does anybody have one or a link to a good example? I tried to ...
- 61
0
votes
0
answers
22
views
PPO on Unity ML Agents, how to discourage truck driving backwards?
In Vector3 (x,y,z) I have access to current road direction (unit vector), velocity, nearest point on road, vectors at whatever offset from the truck's current position I want, etc.
How can I ...
- 574
0
votes
0
answers
20
views
What is an appropriate definition for the action space in my RL problem?
I am thinking about how to define the action space in my RL problem. The goal is to try many different RL algorithms (value-, policy-based, and hybrids), in order to compare their performance, across ...
0
votes
0
answers
40
views
Reinforcement Learning when action space is vectors of percentages summing to 1
I'm writing a reinforcement learning algorithm (PPO) with a continuous action space. In my environment, I have a finite amount of resources that I want to split among 4 different places A, B, C, D. ...
- 101
0
votes
1
answer
291
views
Compatibility of anytrading Gym environment with TF-Agents
All Gym/Gymnasium standard environments are compatible with TwnsorFlow RL agents, but when I tried to use TF-Agents with anytrading I get errors because some required methods and attributes seem to be ...
- 61
0
votes
0
answers
106
views
ModuleNotFoundError: No module named 'gym_anytrading'
Windows 10 operating system, Anaconda used.
import sys
!conda install --yes --prefix {sys.prefix} -c anaconda gymnasium
was successfully completed as well as
<...
- 61
0
votes
0
answers
24
views
Using Discrete class and mask in TF-Agent context for dynamic actions space as a function of the current state
I saw other old posts spinning around this topic, but the general answers given were like "don't care, let the NN learn that, in that specific state, it cannot take some actions punishing it!&...
- 61
1
vote
1
answer
45
views
What is the difference between State Value function and Return for Markov Reward process ( MRP)?
I have been going through Stanford Lecture on RL. I see in MRP that Return function is same as State Value function. Both are getting expected sum of reward keeping discount factor in mind.
Although ...
- 415
0
votes
0
answers
42
views
Do exist Multi-output Deep Reinforcement Learning?
I need a deep reinforcement learning algorithm where the action is an array of N integer values. The idea is that given a state, the action that the agent will decide should be an array of N numbers, ...
- 1
0
votes
0
answers
28
views
Bellman Error for Value Function $V$
I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
0
votes
1
answer
185
views
OpenAI Gym/Gymnasium Custom Env: How should done signal be defined in a continuous task/infinite horizon problem?
I am creating a custom gym environment that should abstract a non-episodic/continuous task.
Generally gym requires to return ...
- 1
0
votes
0
answers
27
views
Problem of extreamly varied reward value in DDQN
I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here.
state shape = 5
action space = 0,1,2, ..., 100
...
- 103
0
votes
0
answers
125
views
Reinforcement Learning (gymnasium's FrozenLake-v1) using Spiking Neural Networks (BindsNet)
I'm new to reinforcement learning. I'm trying to solve the FrozenLake-v1 game using OpenAI's gymnasium learning environment and BindsNet, which is a library to simulate Spiking Neural Networks using ...
- 1
1
vote
1
answer
43
views
In Q-learning, why does Q index on both state and action?
In Q-learning, Q is an array of expected rewards for (state, action) combinations.
It seems to me the same result could be achieved while slightly simplifying the algorithm, if instead of associating ...
- 139
1
vote
1
answer
41
views
Why use sampling instead of the mean value for policy in Reinforcement Learning?
I'm quite new in RL and I'm currently following David Silver's course on RL. But at the same time, I also want to get hands-on, so I followed this tutorial from Gymnasium documentation: https://...
- 13
0
votes
0
answers
24
views
Randomize batches selection on replay_buffer in cartpole game - Tensorflow and RF-agents
I am trying the performances got with different hyperparameters on solving the cartpole Gym environment through TF-Agents using the starting code proposed around the Internet. I copy here the central ...
- 61
4
votes
0
answers
222
views
How Does the Reward Model in ChatGPT Calculate Losses?
Reading the InstructGPT paper(which seems to be what ChatGPT was built off of), I found this equation for the reward function.
However, I'm struggling to understand how this equation is used to ...
- 141
0
votes
0
answers
18
views
Reinforcement Learning - What environment and algorithms should I use?
I have to do a project on Reinforcement Learning.
Environment
First, I need to choose an environment to use. It should meet one of two assumptions:
it should be stochastic OR
it should require ...
- 101
1
vote
1
answer
142
views
In a finite horizon reinforcement learning problem, are the $Q$ and value functions dependent on time?
Typically the definition I see for the $Q$ and value functions is
$$
Q^\pi(s_t, a_t) = \mathbb{E}_\tau\left[\sum_{t'=t}^T\gamma^{t'-t}r(s_{t'}, a_{t'})\ |\ s_t, a_t\right] \\
V^\pi(s_t) = \mathbb{E}_\...
- 113
0
votes
0
answers
18
views
Reinforcement learning discrete action
Assuming a relation such that $y = f(x)$, where $y$ represents a scalar and $x \in 20 \times 1$ vector consisting of zeros and ones, I want to set up a reinforcement learning model that changes the ...
- 101