Questions tagged [reinforcement-learning]
Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.
620
questions
0
votes
0
answers
4
views
Train the transformer with PPO
Context
I'm trying to apply reinforcement learning to the transformer.
I have the following tokens: ...
- 13
0
votes
0
answers
9
views
I use tf where in my rl a2c implementation for action masking. This leads to nan gradients. Can anyone give me a hint how to implement it properly?
right now I work in an research project where we use reinforcement learning.
Because of the big action and state spaces the vanilla A2C doesn`t work well (for this example and my debugging I ...
- 1
0
votes
0
answers
10
views
model suggestions for managing bids on ad platforms
We manage multiple campaigns on platforms like Google Ads and Trivago for our hotels. Part of our job on weekly basis is to adjust bids so our properties win auctions and be visible to guests. Bid ...
- 101
1
vote
0
answers
43
views
reinforcement learning reward choices
To start with, this is not a homework thing. In my attempt to finally get a practical working knowledge of table based re-inforcement learning, I came up with a very silly and easy dice game, serving ...
- 111
0
votes
0
answers
31
views
RL Agent won't learn no matter what I try
I am trying to make an RL model for reinforcement learning, and the environment is fairly simple and a 10x10 grid. During the training phase, it reaches the goal. But when I try to test this, The ...
0
votes
1
answer
49
views
Reinforcement learning for ensemble models
afaik RLHF has been consistently been associated with Gen AI tasks. The reasoning is that since gen AI is stochastic and can generate multiple responses ( based on small changes in prompts, parameters ...
- 342
0
votes
1
answer
96
views
Interpretation of PPO learning curve, value loss, policy loss
my PPO training for a custom gymnasium environment resulted in following outcome. I would need some advice how to interpret the results and where to start activities to improve.
Thank you very much ...
- 61
0
votes
0
answers
10
views
What are the initialization straregies (except random) for the Q-learning algorithm on graphs?
Is there any research / ideas how to initialize Q-function (or DQN) for Q - learning on graphs ? (Except standard random initialization - which seems very far from reasonable one)
Thanks in advance .
0
votes
0
answers
9
views
Reinforcement learning with Q learning doesn't seem to be learning
I'm encountering an issue with my PyTorch-based Q-learning model. Despite implementing the reinforcement learning algorithm, the model seems to be stuck at the same balance level without showing any ...
- 1
0
votes
0
answers
6
views
"Unique" or "Repeated" experiences in memory replay...?
I'm training an RL agent/model (DRL/DQN).
Say that, for each learning step, the memory replay used by the agent to learn, has N elements (experiences) stored, where only X are unique elements (...
0
votes
0
answers
13
views
Reinforcement learning give reward after finishing the data classification instead of acting one by one and CNN based reinforcement learning
I am trying to write an reinforcement based based trading system and while trying to do that, the only way that I can do and I found is rewarding the model for each action but it actually performs bad ...
- 1
0
votes
0
answers
26
views
Using Reinforcement learning for minimisation
I would like to use reinforcement learning for the optimisation of a given function under some contraints.
Take for example the following problems:
...
- 1
0
votes
0
answers
29
views
Combining monte carlo with deep learning to improve the estimation
I am in situation where i need to estimate the attenuation of an EM wave . we consider EM wave as collection of photons. These photons when strike with some dust particles they scatter in different ...
1
vote
1
answer
75
views
In reinforcement learning, why learn Q rather than V?
Why do we learn the action-value function $Q(s,a)$ rather than the (just state) value function $V(s)$? At least for deterministic environments?
$V$ is much smaller than $Q$, and they are trivially ...
- 113
1
vote
0
answers
33
views
Pretrain a Stable Baseline 3 SAC algo from recorded historical / real-life data
I currently have a database recorded from human behavioural (observation, action, reward, next action) with 400k examples collected in real-life conditions.
I want to implement a kind of pretraining ...
- 13
0
votes
1
answer
73
views
Getting actions probabilities instead of an unique prediction in Stable Baselines 3 SAC?
I try to understand how getting an actions probability table instead of an unique prediction in stable baselines 3 SAC in order to override 'predict' method to filter invalid actions. I guess the good ...
- 13
0
votes
0
answers
49
views
Feature extractor for a a 2D array input
I want to train a SAC algorithm from Stable Baselines 3 with a (100,210) shaped array as input. The array is a stack of observations cumulated along axis 0. The last row is current observation.
SAC ...
- 13
0
votes
1
answer
115
views
Illegal action reward strategy for reinforcement learning : reward shaping and termination / truncation
I have some questions about strategy to adopt regarding illagal action handling in reinforcement learning (Stable Baselines 3 / SAC algo). First is about reward shaping, second is about terminating / ...
- 13
0
votes
1
answer
32
views
Learning the gradient descent stepsize with RL [closed]
Problem statement:
I've been working on a project to accelerate the convergence of gradient descent using reinforcement learning (RL). I want to learn a policy that can map the current state of ...
- 1
0
votes
0
answers
68
views
RLHF fine-tune llama2 in vertex ai
I have fine tune RLHF with Vertex AI Pipeline. But deployed model not showing in model registry. Why?
code i have used:
...
1
vote
1
answer
38
views
Why don't search engines filter out unethical/illegal searches?
(Not sure if this question is appropriate to this SE)
I'm studying the LLMs course on Coursera. One topic they deal with is how to get the LLM to not respond with unethical/illegal things, e.g. if you ...
- 275
1
vote
0
answers
35
views
Building an engine for a chess-like game
I'm working on building an AI for a chess-like game. I've implemented a Monty Carlo Tree Search (for the early game) and Rainbow DQN (for the mid to late game), and will be implementing Alpha Beta ...
0
votes
0
answers
13
views
Passing the Parallel API tests in PettingZoo for custom multi-agent environment
from pettingzoo.test import (
parallel_api_test,
parallel_seed_test,
max_cycles_test,
performance_benchmark,
)
I have a custom multiagent ...
- 121
0
votes
0
answers
10
views
DQN with action (obtained from some baseline algorithm) as part of input
I have some base algorithm (that chooses an action and is static but based on rigorous statistical theory). This action is between 0 and 1 and continous,
$$ a_{baseline} \in (0,1) .$$ This means that ...
0
votes
1
answer
50
views
Can I compare Q-VALUE for the same action across different states?
Dears,
I'm new with RL and try to apply to my project.
I've run RL with some example data and got the
My question is if I could compare the Q-values for the same current action across different prior ...
- 1
0
votes
0
answers
161
views
Runtime Error: one of the variables needed for gradient computation has been modified by an inplace operation:
I have the following code for a reinforcement learning using proximal policy optimization. It gives the following run time error.
...
- 37
2
votes
1
answer
64
views
How to represent output layer if the action size changes dynamically?
I am new to ML and I need to train a chess agent using proximal policy optimization. Board is represented as string and the environment gives a list of valid moves for each step.
...
- 37
0
votes
1
answer
32
views
Value Iteration
For value iteration problem, using Bellman equation for iteration 2 , it is clear to me how values are getting calculated.
But for iteration 3 values are unclear to me. Can one help me in this
0
votes
0
answers
63
views
Examples of decision processes where action has no effect on the next state, but has on the reward?
Within my studies, I work on a recurrent reinforcement learning project and I struggle to find real-world problems with a property that is important to my solutions. I look for instances of problems ...
- 660
0
votes
0
answers
16
views
Reinforcement Learning: Formally, does an exponentially decaying epsilon satisfy GLIE?
I am aware that exponentially decaying exploration constants (epsilon) are used practically. Formally, though, do they satisfy the GLIE condition?
Specifically, relating this to the Borel-Cantelli ...
0
votes
0
answers
81
views
Should offline algorithms (SAC, DDGP, TD3) be used with suprocesses vectorized environements?
I read most of offline algorithms from Stable Baselines 3 should not be subprocessed (see here : https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html - "Which algorithm should I ...
- 13
0
votes
0
answers
67
views
What are typical hyperparameter ranges and significance for SB3's offline algorithm?
I'm trying to perform an hyperparameter tuning on a SAC algorithm (Stable Baselines 3) with optuna lib.
I guess hyperparameters have different significance and typical ranges depending of the selected ...
- 13
0
votes
0
answers
39
views
Machine learning can be broken down into supervised, unsupervised, and reinforcement learning. Is there anything else?
Is it even logically possible to have a type of machine learning that doesn't fall into those three paradigms?
Supervised: a dataset of inputs and outputs are fed to an algorithm which learns a ...
0
votes
1
answer
47
views
Reinforcement learning
I am working on a sentiment analysis project. I used BERT model for training but lack of data it gives huge overfitting. So after i moved LLM approach to do that. Using LLM finally i got good results....
0
votes
0
answers
39
views
Bootstrapping formula for TD learning in Reinforcement Learning
In Reinforcement Learning (Sutton & Barto, 2018), p.120, equations (6.3)-(6.4)
, to explain the idea of bootstrapping in Temporal-difference learning:
\begin{equation}
v_{\pi}(s) := E_{\pi}[G_t|...
0
votes
1
answer
71
views
Deep Q-Learning: How are network parameters updated, and why consider episodes in the first place?
I'm trying to wrap my head around the implementation of deep $Q$-learning, and why we even consider episodes in the first place. The usual set-up is that we initialize some starting state $s_0$, then ...
- 101
0
votes
0
answers
62
views
Is n-Step SARSA Off-Policy?
To keep things simple, let us consider the case of an episodic Finite MDP with n=2 and no discounts. Copying the equation for n-step SARSA from Reinforcement Learning 2nd edition, page 146:
The n-step ...
- 11
0
votes
0
answers
40
views
Building an application to group/label browser tabs
I'm trying to develop an application where users can click a button and all of the open tabs in their browser will be placed into tab groupings based on similarity of the tab. Microsoft Edge has a ...
- 1
0
votes
0
answers
21
views
SARSA with Updated Actions
I am trying to gain a better understanding of the Classic SARSA Algorithm. Let us assume Finite MDP which is simple enough so that we can use Tabular methods, so no function approximation. In the ...
- 11
0
votes
0
answers
23
views
Query modification for elastic search using machine learning
I have a problem statement that I'm struggling to formulate as a machine learning framework. There is a huge client database of documents - we're trying to come up with an efficient way of querying ...
- 167
1
vote
1
answer
161
views
Given historical states and action/reward data, is reinforcement learning a reasonable approach?
To summarize my problem:
I want to maximize my total reward over all timesteps
I have 3 discrete actions at each time step.
The state vector for each time step has 5 features. The features are ...
- 251
0
votes
1
answer
424
views
Workflow for reinforcement learning in Python
I'm new to reinforcement learning but (I believe) I have certain experience with machine learning. I know if you work on, say, a classification problem, your workflow will be (in a very high level) ...
-1
votes
0
answers
914
views
Choosing the number of episodes and iterations when training a RL model
I know the parameters chosen for training a RL model depend heavily on the model itself as well as the problem. Nevertheless, I am trying to train a bunch of these agents on different environments, ...
-1
votes
0
answers
24
views
Can somebody explain to me, how the Hyperparameters in this expamle work? I only uderstood the meaning for gamma
I am quite new to this topic, but I want to understand how Reinforcement Learning (RL) works in this example (https://gymnasium.farama.org/tutorials/training_agents/reinforce_invpend_gym_v26/). I have ...
- 1
0
votes
1
answer
52
views
Stick to R or switch to Python as a newby in Reinforcement Learning?
Background
I would consider myself to have a decent level in R and virtually 0 skills in any other programming language - something quite common in my field. Note also that my background is not in ...
- 3
0
votes
0
answers
284
views
Reinforcement learning with PPO - tips and tricks
I am trying to use PPO where the agent has to maneuver around an obstacle towards the target while respecting the spatial boundaries. While the agent learns to respect spatial boundaries it never ...
2
votes
0
answers
21
views
Consistency error in visualization of policy improvement in Sutton & Barto's book?
Sutton & Barto introduce in their foundational book on "Reinforcement Learning: An Introduction" in the context of Dynamic Programming algorithms for policy evaluation and improvement. ...
- 41
0
votes
0
answers
35
views
Pretrained model for RNN Encoder-Decoder?
Our team are implementing a paper called Cold-Start-Reinforcement-Learning-with-Softmax-Policy-Gradient.
Although the paper didn't mention. We want to use a pre-trained model, which is a RNN Encoder-...
- 25
0
votes
1
answer
59
views
Gradients Exploding with Custom Gradient and 2 layer MLP
I am training a 2 layer MLP on an off-policy learning-to-rank task, where the input is a list of documents against a query with a feature vector for each query-document pair, i.e. if there are M ...
- 3,825
0
votes
1
answer
59
views
Inverse reinforcement learning with trajectories only
Inverse reinforcement learning (IRL) is a task that can learn a reward from other agent behaviour. Most IRL paradigms assume that dynamics of environment is known, that is the transition probability ...
- 525