Questions tagged [reinforcement-learning]

Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

Reinforcement Learning: Formally, does an exponentially decaying epsilon satisfy GLIE?

I am aware that exponentially decaying exploration constants (epsilon) are used practically. Formally, though, do they satisfy the GLIE condition? Specifically, relating this to the Borel-Cantelli ...
jeremy.ebg's user avatar
0 votes
0 answers
13 views

Should offline algorithms (SAC, DDGP, TD3) be used with suprocesses vectorized environements?

I read most of offline algorithms from Stable Baselines 3 should not be subprocessed (see here : https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html - "Which algorithm should I ...
GerardL's user avatar
0 votes
0 answers
18 views

What are typical hyperparameter ranges and significance for SB3's offline algorithm?

I'm trying to perform an hyperparameter tuning on a SAC algorithm (Stable Baselines 3) with optuna lib. I guess hyperparameters have different significance and typical ranges depending of the selected ...
GerardL's user avatar
0 votes
0 answers
30 views

Machine learning can be broken down into supervised, unsupervised, and reinforcement learning. Is there anything else?

Is it even logically possible to have a type of machine learning that doesn't fall into those three paradigms? Supervised: a dataset of inputs and outputs are fed to an algorithm which learns a ...
Austin Capobianco's user avatar
0 votes
1 answer
36 views

Reinforcement learning

I am working on a sentiment analysis project. I used BERT model for training but lack of data it gives huge overfitting. So after i moved LLM approach to do that. Using LLM finally i got good results....
Sandun Tharaka's user avatar
0 votes
0 answers
17 views

Bootstrapping formula for TD learning in Reinforcement Learning

In Reinforcement Learning (Sutton & Barto, 2018), p.120, equations (6.3)-(6.4) , to explain the idea of bootstrapping in Temporal-difference learning: \begin{equation} v_{\pi}(s) := E_{\pi}[G_t|...
fermented_bean's user avatar
0 votes
1 answer
53 views

Deep Q-Learning: How are network parameters updated, and why consider episodes in the first place?

I'm trying to wrap my head around the implementation of deep $Q$-learning, and why we even consider episodes in the first place. The usual set-up is that we initialize some starting state $s_0$, then ...
infinitylord's user avatar
0 votes
0 answers
19 views

Is n-Step SARSA Off-Policy?

To keep things simple, let us consider the case of an episodic Finite MDP with n=2 and no discounts. Copying the equation for n-step SARSA from Reinforcement Learning 2nd edition, page 146: The n-step ...
mashrivas's user avatar
0 votes
0 answers
18 views

Building an application to group/label browser tabs

I'm trying to develop an application where users can click a button and all of the open tabs in their browser will be placed into tab groupings based on similarity of the tab. Microsoft Edge has a ...
Ben's user avatar
  • 1
0 votes
0 answers
17 views

SARSA with Updated Actions

I am trying to gain a better understanding of the Classic SARSA Algorithm. Let us assume Finite MDP which is simple enough so that we can use Tabular methods, so no function approximation. In the ...
mashrivas's user avatar
0 votes
0 answers
19 views

Query modification for elastic search using machine learning

I have a problem statement that I'm struggling to formulate as a machine learning framework. There is a huge client database of documents - we're trying to come up with an efficient way of querying ...
user9343456's user avatar
1 vote
1 answer
141 views

Given historical states and action/reward data, is reinforcement learning a reasonable approach?

To summarize my problem: I want to maximize my total reward over all timesteps I have 3 discrete actions at each time step. The state vector for each time step has 5 features. The features are ...
dcl's user avatar
  • 251
0 votes
1 answer
70 views

Workflow for reinforcement learning in Python

I'm new to reinforcement learning but (I believe) I have certain experience with machine learning. I know if you work on, say, a classification problem, your workflow will be (in a very high level) ...
ElonMuskofBadIdeas's user avatar
0 votes
0 answers
184 views

Choosing the number of episodes and iterations when training a RL model

I know the parameters chosen for training a RL model depend heavily on the model itself as well as the problem. Nevertheless, I am trying to train a bunch of these agents on different environments, ...
user152104's user avatar
0 votes
0 answers
22 views

Can somebody explain to me, how the Hyperparameters in this expamle work? I only uderstood the meaning for gamma

I am quite new to this topic, but I want to understand how Reinforcement Learning (RL) works in this example (https://gymnasium.farama.org/tutorials/training_agents/reinforce_invpend_gym_v26/). I have ...
Peter's user avatar
  • 1
0 votes
1 answer
38 views

Stick to R or switch to Python as a newby in Reinforcement Learning?

Background I would consider myself to have a decent level in R and virtually 0 skills in any other programming language - something quite common in my field. Note also that my background is not in ...
ionatura's user avatar
0 votes
0 answers
79 views

Reinforcement learning with PPO - tips and tricks

I am trying to use PPO where the agent has to maneuver around an obstacle towards the target while respecting the spatial boundaries. While the agent learns to respect spatial boundaries it never ...
SathukaBootham's user avatar
2 votes
0 answers
15 views

Consistency error in visualization of policy improvement in Sutton & Barto's book?

Sutton & Barto introduce in their foundational book on "Reinforcement Learning: An Introduction" in the context of Dynamic Programming algorithms for policy evaluation and improvement. ...
Steve's user avatar
  • 41
0 votes
0 answers
24 views

Pretrained model for RNN Encoder-Decoder?

Our team are implementing a paper called Cold-Start-Reinforcement-Learning-with-Softmax-Policy-Gradient. Although the paper didn't mention. We want to use a pre-trained model, which is a RNN Encoder-...
jackson's user avatar
  • 25
0 votes
1 answer
42 views

Gradients Exploding with Custom Gradient and 2 layer MLP

I am training a 2 layer MLP on an off-policy learning-to-rank task, where the input is a list of documents against a query with a feature vector for each query-document pair, i.e. if there are M ...
SHASHANK GUPTA's user avatar
0 votes
1 answer
25 views

Inverse reinforcement learning with trajectories only

Inverse reinforcement learning (IRL) is a task that can learn a reward from other agent behaviour. Most IRL paradigms assume that dynamics of environment is known, that is the transition probability ...
JunjieChen's user avatar
0 votes
0 answers
26 views

Is it true that a multi-armed-bandit selects an action based on the future reward?

During my machine learning lecture I saw the following statement on the slides: "Multi-armed-bandit selects an action based on the future reward" Is that statement true? In my opinion it is ...
user150593's user avatar
0 votes
0 answers
18 views

Dueling DQN with varying number of actions

I have an RL problem, where the number of actions depends on the state. Furthermore, each action-value computation requires action information in the form of a high-dimensional, continuous vector in ...
WolfSovereign's user avatar
0 votes
0 answers
24 views

State and next state is the same on openai gym atari

i am using OpenGym atari to trainning my Pacman agent, this is part of my code ...
tuyenhx's user avatar
0 votes
0 answers
31 views

find q-table for discrete action space

I am trying to use q-learning for a discrete observation space that is represented by: buffer: list of 200 integer values in [0,10] discard_counter: list of 200 integer values in [0, 4] capacity: ...
Jannis's user avatar
  • 1
0 votes
0 answers
16 views

Multiagent observations encoding - LSTM or Transformer?

In multi agent learning problems where agents can join or leave, to address the issue of varying observation space I am trying to build a latent representation either using LSTM or transformer which ...
SathukaBootham's user avatar
0 votes
0 answers
14 views

Policy Gradient training log-derivative un-normalized vs normalized objective

I am implementing a policy gradient training objective for optimizing ranking metrics in a learning-to-rank setting. For a given query $q$, a set of documents $D_q$ (retrieved from a first-stage ...
SHASHANK GUPTA's user avatar
0 votes
0 answers
23 views

SB3-PPO and Keras-PPO on CartPole env: performances can be improved and catastrophic forgetting limited?

I am trying to experiment different hyperparameters and neural network structures in order to stabilize the learning process of the CartPole environment and possibly reduce the occurrence and gravity ...
fede72bari's user avatar
0 votes
0 answers
14 views

Llimit/masking the space of actions to the legal ones only: A -> A(S)

I have difficulties on coding a way to limit the available actions as a function of the current agent state S. I am trying to use https://keras.io/examples/rl/ppo_cartpole/ (that works quite fine) and ...
fede72bari's user avatar
1 vote
0 answers
51 views

A working tf_agents.agents.PPOClipAgent example for reference?

I am really struggling on finding a working example for tf_agents.agents.PPOClipAgent, for instance for the simple CartPoleV1 environment. Does anybody have one or a link to a good example? I tried to ...
fede72bari's user avatar
0 votes
0 answers
22 views

PPO on Unity ML Agents, how to discourage truck driving backwards?

In Vector3 (x,y,z) I have access to current road direction (unit vector), velocity, nearest point on road, vectors at whatever offset from the truck's current position I want, etc. How can I ...
mLstudent33's user avatar
0 votes
0 answers
20 views

What is an appropriate definition for the action space in my RL problem?

I am thinking about how to define the action space in my RL problem. The goal is to try many different RL algorithms (value-, policy-based, and hybrids), in order to compare their performance, across ...
WolfSovereign's user avatar
0 votes
0 answers
39 views

Reinforcement Learning when action space is vectors of percentages summing to 1

I'm writing a reinforcement learning algorithm (PPO) with a continuous action space. In my environment, I have a finite amount of resources that I want to split among 4 different places A, B, C, D. ...
Aydin Abiar's user avatar
0 votes
1 answer
289 views

Compatibility of anytrading Gym environment with TF-Agents

All Gym/Gymnasium standard environments are compatible with TwnsorFlow RL agents, but when I tried to use TF-Agents with anytrading I get errors because some required methods and attributes seem to be ...
fede72bari's user avatar
0 votes
0 answers
106 views

ModuleNotFoundError: No module named 'gym_anytrading'

Windows 10 operating system, Anaconda used. import sys !conda install --yes --prefix {sys.prefix} -c anaconda gymnasium was successfully completed as well as <...
fede72bari's user avatar
0 votes
0 answers
24 views

Using Discrete class and mask in TF-Agent context for dynamic actions space as a function of the current state

I saw other old posts spinning around this topic, but the general answers given were like "don't care, let the NN learn that, in that specific state, it cannot take some actions punishing it!&...
fede72bari's user avatar
1 vote
1 answer
44 views

What is the difference between State Value function and Return for Markov Reward process ( MRP)?

I have been going through Stanford Lecture on RL. I see in MRP that Return function is same as State Value function. Both are getting expected sum of reward keeping discount factor in mind. Although ...
Arpit Sisodia's user avatar
0 votes
0 answers
42 views

Do exist Multi-output Deep Reinforcement Learning?

I need a deep reinforcement learning algorithm where the action is an array of N integer values. The idea is that given a state, the action that the agent will decide should be an array of N numbers, ...
Floder's user avatar
  • 1
0 votes
0 answers
28 views

Bellman Error for Value Function $V$

I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
Vera Leighton 's user avatar
0 votes
1 answer
183 views

OpenAI Gym/Gymnasium Custom Env: How should done signal be defined in a continuous task/infinite horizon problem?

I am creating a custom gym environment that should abstract a non-episodic/continuous task. Generally gym requires to return ...
94621's user avatar
  • 1
0 votes
0 answers
27 views

Problem of extreamly varied reward value in DDQN

I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here. state shape = 5 action space = 0,1,2, ..., 100 ...
Subhajit Saha's user avatar
0 votes
0 answers
125 views

Reinforcement Learning (gymnasium's FrozenLake-v1) using Spiking Neural Networks (BindsNet)

I'm new to reinforcement learning. I'm trying to solve the FrozenLake-v1 game using OpenAI's gymnasium learning environment and BindsNet, which is a library to simulate Spiking Neural Networks using ...
gthampi's user avatar
1 vote
1 answer
43 views

In Q-learning, why does Q index on both state and action?

In Q-learning, Q is an array of expected rewards for (state, action) combinations. It seems to me the same result could be achieved while slightly simplifying the algorithm, if instead of associating ...
rwallace's user avatar
  • 139
1 vote
1 answer
41 views

Why use sampling instead of the mean value for policy in Reinforcement Learning?

I'm quite new in RL and I'm currently following David Silver's course on RL. But at the same time, I also want to get hands-on, so I followed this tutorial from Gymnasium documentation: https://...
GA Faza's user avatar
  • 13
0 votes
0 answers
24 views

Randomize batches selection on replay_buffer in cartpole game - Tensorflow and RF-agents

I am trying the performances got with different hyperparameters on solving the cartpole Gym environment through TF-Agents using the starting code proposed around the Internet. I copy here the central ...
fede72bari's user avatar
4 votes
0 answers
219 views

How Does the Reward Model in ChatGPT Calculate Losses?

Reading the InstructGPT paper(which seems to be what ChatGPT was built off of), I found this equation for the reward function. However, I'm struggling to understand how this equation is used to ...
itisyeetimetoday's user avatar
0 votes
0 answers
18 views

Reinforcement Learning - What environment and algorithms should I use?

I have to do a project on Reinforcement Learning. Environment First, I need to choose an environment to use. It should meet one of two assumptions: it should be stochastic OR it should require ...
nietoperz21's user avatar
1 vote
1 answer
139 views

In a finite horizon reinforcement learning problem, are the $Q$ and value functions dependent on time?

Typically the definition I see for the $Q$ and value functions is $$ Q^\pi(s_t, a_t) = \mathbb{E}_\tau\left[\sum_{t'=t}^T\gamma^{t'-t}r(s_{t'}, a_{t'})\ |\ s_t, a_t\right] \\ V^\pi(s_t) = \mathbb{E}_\...
user3002473's user avatar
0 votes
0 answers
18 views

Reinforcement learning discrete action

Assuming a relation such that $y = f(x)$, where $y$ represents a scalar and $x \in 20 \times 1$ vector consisting of zeros and ones, I want to set up a reinforcement learning model that changes the ...
user135567's user avatar
0 votes
0 answers
22 views

Using RecSim trained agent in production

I am new to using Reinforcement Learning in Recommender Systems. Can someone please give me pointers on how to use an agent trained using Google's Recsim in production?
user882763's user avatar

1
2 3 4 5
13