Questions tagged [reinforcement-learning]
Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.
619
questions
1
vote
1
answer
1k
views
Reinforcement Learning - PPO: Why do so many implementations calculate the returns using the GAE? (Mathematical reason)
There are so many PPO implementations that use GAE and do the following:
...
CommunityBot
- 1
0
votes
0
answers
15
views
Automated detection between single and multi agent RL algo [closed]
I am not sure if this post belongs here, please let me know if it doesn't. I have multiple RL scripts of both single agent (PPO, DQGN), and multi agent(MADDPG, MAA2C). I am looking to build a ...
2
votes
2
answers
116
views
Idenitity between TD(0) algorithm and Policy Evaluation in Dynamic Programming when alpha is equal to 1
TD(0) algorithm is defined as the iterative update of the following:
$$ V(s) \leftarrow V(s) + \alpha({r + \gamma V(s')} - V(s) ) $$
Now, if we assume alpha to be equal to 1, we get the traditional ...
CommunityBot
- 1
3
votes
1
answer
61
views
What's the definition of retraining？
In transfer learning, we always use new data to retrain the pre-trained model. But, what is the specific and official definition of retraining? Or what papers mentioned this definition, in transfer ...
CommunityBot
- 1
1
vote
1
answer
164
views
experience replay memory: saving the next state required when state does not depend on action?
so, I am using an agent with a state-action-policy and I am trying to understand the concept of experience replay memory (ERM). As far as I learned until now, the ERM is basically a buffer that stores ...
CommunityBot
- 1
1
vote
2
answers
750
views
How to calculate Temperature variable in softmax(boltzmann) exploration
Hi I am developing a reinforcement learning agent for a continous state/discrete action space. I am trying to use boltmzann/softmax exploration as action selection strategy. My action space is of size ...
CommunityBot
- 1
2
votes
1
answer
214
views
Policy gradient - and auto-differentiation (Pytorch/Tensorflow)
In policy gradient, we have something like this:
Is my understanding correct that if I apply log cross-entropy on the last layer, the gradient will be automatically calculated as per formula above?
CommunityBot
- 1
0
votes
0
answers
20
views
Using Reinforcement learning for minimisation
I would like to use reinforcement learning for the optimisation of a given function under some contraints.
Take for example the following problems:
...
- 1
3
votes
3
answers
230
views
Is there a rule of thumb when designing neural network in deep reinforcement learning?
In deep learning, we can assess model's performance with loss function value and improve model's performance with K-fold cross-validation and so on. But how can we design and tune neural network used ...
CommunityBot
- 1
0
votes
0
answers
24
views
Combining monte carlo with deep learning to improve the estimation
I am in situation where i need to estimate the attenuation of an EM wave . we consider EM wave as collection of photons. These photons when strike with some dust particles they scatter in different ...
1
vote
1
answer
36
views
In reinforcement learning, why learn Q rather than V?
Why do we learn the action-value function $Q(s,a)$ rather than the (just state) value function $V(s)$? At least for deterministic environments?
$V$ is much smaller than $Q$, and they are trivially ...
- 1,045
1
vote
1
answer
81
views
Is reinforcement learning suitable for the Dial-a-Ride problem?
Is reinforcement learning suitable for this problem or will it perform poorly against classical algorithms?
"The Dial-a-Ride Problem (DARP) consists of designing vehicle routes and schedules for ...
CommunityBot
- 1
1
vote
1
answer
197
views
Why not use max(returns) instead of average(returns) in off-policy Monte Carlo control?
As I understand it, in reinforcement learning, off-policy Monte Carlo control is when the state-action value function $Q(s,a)$ is estimated as a weighted average of the observed returns.
However, in ...
CommunityBot
- 1
0
votes
1
answer
1k
views
gym car racing v0 using DQN
I am currently learning reinforcement learning and wanted to use it on the car racing-v0 environment. I have successfully made it using PPO algorithm and now I want to use a DQN algorithm but when I ...
CommunityBot
- 1
1
vote
1
answer
119
views
Policy Gradient custom loss function not working
I was experimenting with my policy gradient reinforcement learning algorithm, and I was wondering if I could use a similar method to the supervised cross-entropy. So, instead of using existing labels, ...
CommunityBot
- 1
1
vote
0
answers
21
views
Pretrain a Stable Baseline 3 SAC algo from recorded historical / real-life data
I currently have a database recorded from human behavioural (observation, action, reward, next action) with 400k examples collected in real-life conditions.
I want to implement a kind of pretraining ...
- 13
8
votes
1
answer
286
views
How to predict advantage value in deep reinforcement learning
I'm currently working on a collection of reinforcement algorithms: https://github.com/lhk/rl_gym
For deep q-learning, you need to calculate the q-values that should be predicted by your network. There ...
CommunityBot
- 1
1
vote
1
answer
410
views
Effects of slipperiness in OpenAI FrozenLake Environment
I am trying to wrap my head around the effects of is_slippery in the open.ai FrozenLake-v0 environment.
From my results when ...
CommunityBot
- 1
0
votes
1
answer
153
views
Reinforcement Learning in a game against itself?
Let's we have a tictactoe design using RL against a random player. We can describe the system by enhancing and giving rewards to good actions. But what if the Rl model is played with itself? What ...
CommunityBot
- 1
1
vote
1
answer
72
views
What is the difference between State Value function and Return for Markov Reward process ( MRP)?
I have been going through Stanford Lecture on RL. I see in MRP that Return function is same as State Value function. Both are getting expected sum of reward keeping discount factor in mind.
Although ...
CommunityBot
- 1
0
votes
0
answers
24
views
Getting actions probabilities instead of an unique prediction in Stable Baselines 3 SAC?
I try to understand how getting an actions probability table instead of an unique prediction in stable baselines 3 SAC in order to override 'predict' method to filter invalid actions. I guess the good ...
- 13
3
votes
1
answer
69
views
How can I train a model to modify a vector by rewarding the model based on the modified vectors nearest neighbors?
I am experimenting with a document retrieval system in which I have documents represented as vectors. When queries come in, they are turned to vectors by the same method as used for the documents. The ...
CommunityBot
- 1
0
votes
1
answer
59
views
Illegal action reward strategy for reinforcement learning : reward shaping and termination / truncation
I have some questions about strategy to adopt regarding illagal action handling in reinforcement learning (Stable Baselines 3 / SAC algo). First is about reward shaping, second is about terminating / ...
- 393
3
votes
1
answer
291
views
Policy Gradient not "learning"
I'm attempting to implement the policy gradient taken from the "Hands-On Machine Learning" book by Geron, which can be found here. The notebook uses Tensorflow and I'm attempting to do it with PyTorch....
CommunityBot
- 1
0
votes
2
answers
173
views
Reinforcement Learning in NLP for chatbots
Is anyone aware of any successful implementation of reinforcement learning for NLP. I am looking to for chatbots which can learn automatically.
Tried searching internet but found very few articles ...
CommunityBot
- 1
1
vote
1
answer
600
views
Difference between Q-learning and G-learning in Reinforcement Learning?
What is the difference between Q-learning and G-learning in Reinforcement Learning? Please explain with formulas.
An example source:
Instead of relying on a utility of consumption, we present G-...
CommunityBot
- 1
0
votes
0
answers
46
views
Feature extractor for a a 2D array input
I want to train a SAC algorithm from Stable Baselines 3 with a (100,210) shaped array as input. The array is a stack of observations cumulated along axis 0. The last row is current observation.
SAC ...
- 1,988
3
votes
3
answers
625
views
Is reinforcement learning a subset of unsupervised learning?
According to this article:
Reinforcement learning on the other hand, which is a subset of Unsupervised learning ...
How true is this statement? Is there any scholarly discussion/writing on the ...
- 101
2
votes
1
answer
1k
views
inverted pendulum REINFORCE
I am learning reinforcement learning, and as a practice, I am trying to stabilize an inverted pendulum (gym: Pendulum-v0) in an upright position using policy gradient: REINFORCE.
I have some ...
CommunityBot
- 1
0
votes
1
answer
397
views
Compatibility of anytrading Gym environment with TF-Agents
All Gym/Gymnasium standard environments are compatible with TwnsorFlow RL agents, but when I tried to use TF-Agents with anytrading I get errors because some required methods and attributes seem to be ...
CommunityBot
- 1
2
votes
1
answer
274
views
How does Q-Learning deal with mixed strategies?
I'm trying to understand how Q-learning deals with games where the optimal policy is a mixed strategy. The Bellman equation says that you should choose $max_a(Q(s,a))$ but this implies a single unique ...
CommunityBot
- 1
1
vote
1
answer
699
views
Reinforcement learning: negative reward (punish) illegal actions?
If you train an agent using reinforcement learning (with Q-function in this case), should you give a negative reward (punish) if the agent proposes illegal actions for the presented state?
I guess ...
CommunityBot
- 1
1
vote
2
answers
113
views
Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?
The course of David Silver about Reinforcement Learning explains how you get the optimal policy from the optimal value function.
It seems to be very simple, you just have to act greedily, by ...
CommunityBot
- 1
2
votes
1
answer
348
views
Problem when cherry picking actions - Proximal Policy Optimization
I am using the implementation of PPO2 in stable-baselines (a fork of OpenAI's baselines) for a Reinforcement Learning problem.
My observation space is $9x9x191$ and my action space is $144$. Given a ...
CommunityBot
- 1
0
votes
1
answer
86
views
Deep RL: How often should retraining be done?
As the headline suggests, how often should retraining be performed when using deep RL? I guess retraining after every action is too expensive? I also guess there is no specific number (e.g. after 1,...
CommunityBot
- 1
2
votes
1
answer
135
views
How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )
How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )?
I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear ...
CommunityBot
- 1
1
vote
1
answer
180
views
Cartpole - Number of layers and neurons - model hyperparameters
Can anyone please suggest me how to arrive to the best optimal values for number of layers, number of neurons parameters of the deep learning model in DDQN algorithm for cartpole problem.
As input and ...
- 21.2k
3
votes
2
answers
626
views
Reward function to avoid illegal actions, minimize legal action and learn to win - Reinforcement Learning
I'm currently implementing PPO for a game with the following characteristics:
Observation space: 9x9x(>150)
Action space: 144
In a given state, only a handful of actions (~1-10) are legal
The state ...
CommunityBot
- 1
4
votes
1
answer
1k
views
DQN fails to find optimal policy
Based on DeepMind publication, I've recreated the environment and I am trying to make the DQN find and converge to an optimal policy. The task of an agent is to learn how to sustainably collect apples ...
CommunityBot
- 1
2
votes
1
answer
359
views
Policy gradient/REINFORCE algorithm with RNN: why does this converge with SGM but not Adam?
I am working on training RNN model on caption generation with REINFORCE algorithm. I adopt self-critic strategy (see paper Self-critical Sequence Training for Image Captioning) to reduce the variance. ...
CommunityBot
- 1
3
votes
1
answer
522
views
Choosing a right algorithm for template-based text generation
I am doing a text generation project -- the task is to basically represent the statistical data in a readable way.
The way I decided to go about this is template-based: each data type has a template ...
CommunityBot
- 1
0
votes
1
answer
25
views
Learning the gradient descent stepsize with RL [closed]
Problem statement:
I've been working on a project to accelerate the convergence of gradient descent using reinforcement learning (RL). I want to learn a policy that can map the current state of ...
- 540
4
votes
1
answer
430
views
Time horizon T in policy gradients (actor-critic)
I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture.
At the bottom of that slide, the gradient of the expected sum of rewards ...
CommunityBot
- 1
0
votes
0
answers
47
views
RLHF fine-tune llama2 in vertex ai
I have fine tune RLHF with Vertex AI Pipeline. But deployed model not showing in model registry. Why?
code i have used:
...
1
vote
1
answer
31
views
Why don't search engines filter out unethical/illegal searches?
(Not sure if this question is appropriate to this SE)
I'm studying the LLMs course on Coursera. One topic they deal with is how to get the LLM to not respond with unethical/illegal things, e.g. if you ...
- 36
2
votes
1
answer
165
views
Reinforcement Learning on real time data over a web server
Question: is it possible to implement a reinforcement learning model over a NodeJS server?
This server would be receiving binary forms of data (open /close; yes/no) in real time.
The objective for ...
CommunityBot
- 1
2
votes
1
answer
92
views
What is the best way Reinforcement learning, RNN or others to predict the best action we have to take to maximize sales?
I have a dataset composed of few features :
customerId, actionDay1, SalesDay1, actionDay20, SalesDay20, actionDay30, SalesDay30
action can be :
call
email
face ...
CommunityBot
- 1
1
vote
1
answer
3k
views
IndexError: index 804 is out of bounds for axis 0 with size 800
i installed a self driving car project from superdatascience site , when i open the map using terminal after a while the map window close up or it closes directly after i maximize the map window and ...
CommunityBot
- 1
0
votes
2
answers
2k
views
Understanding action space in stable baselines
I was trying to write reinforcement learning agent using stable-baselines3 library. The agent(abservations) method should return action. I went through different ...
CommunityBot
- 1
0
votes
1
answer
192
views
Reinforcement Learning model always gives different output
I am trying to build a reinforcement learning model for hardware capacity optimisation. The state of the model would input like CPU capacity utilisation, memory utilisation. The model is supposed to ...
CommunityBot
- 1