As of May 31, 2023, we have updated our Code of Conduct.

Questions tagged [markov-process]

A Markov process is a stochastic process for which the Markov property holds: If you know the current state, then the next state is independent of all past states.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

For a node in an undirected graph - does the node affect itself if its markov blanket is known?

Consider the following Markov Random Field. Question 1: Which of the following nodes will have no effect on H given the Markov Blanket of H? Question 2: Will node H itself have any effect on itself, ...
Deepak Tatyaji Ahire's user avatar
0 votes
0 answers
11 views

Is the node in undirected graph itself included in the set of its own Markov Blanket?

Consider an undirected graph with nodes set {a,b,c,d,e}, and edge set {(a,b), (a,c), (a,d), (a,e)}. From the above info, you will clearly visualise that the node a ...
Deepak Tatyaji Ahire's user avatar
1 vote
1 answer
23 views

What is the difference between State Value function and Return for Markov Reward process ( MRP)?

I have been going through Stanford Lecture on RL. I see in MRP that Return function is same as State Value function. Both are getting expected sum of reward keeping discount factor in mind. Although ...
Arpit Sisodia's user avatar
0 votes
0 answers
8 views

Attribution modelling using First and Higher-Order Markov Chains

The crux of my question is as follows: Would a higher-order Markov model produce a different result than a first-order Markov model when used for Channel Attribution modelling? Once the transition ...
Aditya Kulkarni's user avatar
0 votes
0 answers
12 views

Find transistion matrix Markov chain

There is a two-state discrete-time Markov chain with a random variable: $y_t = yx_t$ where $y = [1 \ 5]'$ (' is there because this matrix should be transposed). It is known that: $E(y_{t+1} | x_t) = [...
Ирина Мухомор's user avatar
0 votes
0 answers
39 views

How should I fit and transform markov transition field in this time series?

I have a group of 900 samples each with 100 time steps and 5 features, and 1 label for each sample (900). I want to image that time series with MarkovTransitionField pyts to then make classification. ...
user9085964's user avatar
0 votes
0 answers
12 views

word2vec, attention for predicting the next state in business processes

For this repo and paper: https://github.com/diogoff/unlabelled-event-logs Business processes are modeled as Markov and Expectation Maximization is used to find the model. So suppose a business process ...
mLstudent33's user avatar
1 vote
1 answer
62 views

memory error- python N-th order Markovian transition matrix from a given sequence

Ok. What is wrong with you code! I am trying to calculate transition probabilities for each leg. The code works for small array but for the actual dataset I got memory error. I have 64 g version ...
Dr. Turkuaz's user avatar
1 vote
1 answer
531 views

Markov Process and transition matrix

I would like to find some good courses but also a quick response on how to model transition matrix given the states. Imagine having 4 states and the following array [1,2,4,1,3,4,2 etc etc]. What ...
minattosama's user avatar
1 vote
0 answers
61 views

Jacks car rental problem: why deterministic policies?

In Sutton & Barto Book: Reinforcement Learning: An Introduction, there is the following problem: I have this question: why are the policies to be considered here are deterministic?
data_science_learner's user avatar
3 votes
2 answers
1k views

Can Reinforcement Learning learn to be deceptive?

I have seen several exampled of deploying RL agents in deceptive environnement or games and the agent learns to perform its tasks regardless. What about the other way around? Can RL be used to create ...
NoRel's user avatar
  • 47
0 votes
1 answer
87 views

Suggestions for studying *Clickstream* data

I've essentially been handed a dataset of website access history and I'm trying to draw some conclusions from it. The data supplied gives me the web URL, the datetime for when it was accessed, an the ...
user1147964's user avatar
1 vote
1 answer
133 views

Markov Chain vs Bayes Net

I am learning about Markov Chain and Bayesian Nets. However at this point I am a bit confused about what types of problems are modelled with the two different models presented to us. From what I ...
JANVI SHARMA's user avatar
2 votes
0 answers
29 views

"Learning" algorithm to use when future depends on past events (MDP property not met)

There are around 5 different retirement plans available in my country. People can pick from them freely. I would like to create a solution that would try to predict the best plan(s) given a particular ...
White_Raven's user avatar
0 votes
1 answer
460 views

Difference between $Q(s,a)$ ,$V^*(s)$ and $V^\pi(s)$ in Markov Decision Process?

I am new to RL and I am trying to understand how to find solutions of an MDP. This is what I understand so far -> since the nature of our environment is stochastic, at a state 's' if we take an ...
JANVI SHARMA's user avatar
1 vote
0 answers
24 views

global independence vs local independence in markov network

I could not understand the local independence and global independence of a markov network. Please help me understand with a simple graph
prog's user avatar
  • 135
1 vote
2 answers
74 views

Choosing a model for input: categorised, weighted sequence, output: binary variable

What would be an appropriate model for predicting a binary target variable, given a weighted sequence? Sequences will be reasonably short, typically between ~ 1 and 5 elements. Illustrated example Say ...
Ian's user avatar
  • 23
1 vote
0 answers
45 views

state transition classification on terminal state

I have data on a unit $i$ which enters an entry state $S_0$. This unit has some covariates $x_i$ I would like to predict the probability the unit will reach the terminal state $S_{pos}$ or $S_{neg}$. ...
Hanan Shteingart's user avatar
1 vote
0 answers
74 views

How do I test and validate a Markov Model in RStudio?

I've created a Markov Model in RStudio using the seqHMM library. I'm new to R and would usually do this kind of thing in Python, but I wanted to use seqHMM. I can see the model has been created and it ...
logic-unit's user avatar
2 votes
1 answer
58 views

Regime detection to identify transitions between habitats

The following figure represents the concentration of a substance (referred to as Element in the code) measured in an organism throughout its life. There are ...
Ryan's user avatar
  • 125
1 vote
0 answers
79 views

Ranking graph's nodes by score propagation

Problem I have the following directed tripartite graph $G(E\cup V\cup P, A)$, where there is a many-to-one symmetric relationship between the subsets V and E - $e\in E,v\in V,[e, v]\in A \iff [v, e]\...
hldev's user avatar
  • 111
1 vote
0 answers
28 views

Hidden Markov Model with Autoregressive emission model?

So far, all standard HMM implementations I've seen assume some variation of a Gaussian Mixture (GMM) as their emission model. It can of course only have a single mixture component which reduces it to ...
user3641187's user avatar
1 vote
0 answers
35 views

Estimating model for transition probabilities of a Markov Chain

Suppose that I have a Markov chain with $S$ states evolving over time. I have $S^2\times T$ values of the transition matrix, where $T$ is the number of time periods. I also have $K$ matrices $X$ of $T\...
DanielTheRocketMan's user avatar
2 votes
1 answer
62 views

Markov Decision Process representation

I'm attempting to model a simple process using a Markov Decision Process. Let $A$ be a set of $3$ actions : $ A \in \{b,s\}$. $T(s,a,s')$ represents the probability of if in state $s$ , take action $...
blue-sky's user avatar
  • 207
3 votes
1 answer
51 views

If I use Gibbs sampling with a Bayesian model, what do I have to check is memoryless?

Right now I am trying to better understand how Bayesian modeling works with just the basics. I found through reading tutorials that some very basic Bayesian models like Bayesian Hierarchical Modeling ...
pierround's user avatar
0 votes
1 answer
34 views

Find changes in variables into two states

I have a dataframe like this: ...
Nathalie's user avatar
  • 147
0 votes
0 answers
23 views

Machine Learning alternative for hashing

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
user3078335's user avatar
0 votes
1 answer
232 views

Best python library for training using Hidden Marov model with Gaussian Mixture

I would like to train my data using HMM- GMM (Baum Welch approach with gaussian Mixture) to find the best parameters suited for my data. Note : My data is ...
Mari's user avatar
  • 165
1 vote
1 answer
738 views

Q-learning when minimising a total cost instead of maximising a total reward

I have a decision problem where the results are measured as a cost that I want to minimise. It seems like a good fit to Q-learning, but I am not sure how to adjust it to deal with a cost instead of a ...
EArwa's user avatar
  • 75
1 vote
2 answers
110 views

Machine Learning algorithm for detecting anomalies in large sets of events

Let's start with the following hypothetical preconditions: There is traffic: normal and anomaly. Each traffic sample contains a list of events (of variable size) Events happen in order, the possible ...
Denis Rozimovschii's user avatar
0 votes
1 answer
65 views

Reinforcement Learning control with known dynamic equation

I know there is model-based reinforcement learning. But all the approaches assume an MDP. If I want to do a feedback control of a system (i. e. control an inverted pendulum) it's quite easy to find ...
aacgymag's user avatar
1 vote
1 answer
338 views

Predict how many days late or early someone will finish their work

So I have a set of deadlines and people, with a database of when those people finished their previous work and how much after the deadline it was, as well as when the work was given. The work itself ...
GenRincewind's user avatar
4 votes
1 answer
39 views

Artificially increasing frequency weight of word ending characters in word building

I have a database of letter pair bigrams. For example: ...
Matt's user avatar
  • 141
1 vote
0 answers
81 views

Reinforcement learning - generating a matrix of continuous values with varying size for test data generation

Currently, I am using RL A3C algorithm for test data generation, where for a set of 30 functions written in C (mostly basic algorithms like Prime number checks, triangle validity, etc.) I try to ...
Cegy's user avatar
  • 21
1 vote
1 answer
137 views

Evaluating value functions in RL

I'm working my way through the book Reinforcement Learning by Richar S. Sutton and Andrew G. Barto and I am stuck on the following question. The value of a state depends on the the values of the ...
buydadip's user avatar
  • 179
4 votes
1 answer
752 views

Reinforcement Learning - How are these state values in MRP calculated?

This is a question from the book an Introduction to RL, page 125, example 6.2. The example compares the prediction abilities of TD(0) and constant $ \alpha $ MC when applied to the below Markov ...
Melanie A's user avatar
  • 167
2 votes
2 answers
1k views

What are the differences between Reinforcement Learning (RL) and Supervised Learning?

What is the difference between Reinforcement Learning (RL) and Supervised Learning? Does RL hava more difficulty in finding a stable solution? Does Q-learning have more difficulty in finding a ...
user10296606's user avatar
  • 1,754
4 votes
1 answer
9k views

What is the relationship between MDP and RL?

What is the relationship between Markov Decision Processes and Reinforcement Learning? Could we say RL and DP are two types of MDP?
user10296606's user avatar
  • 1,754
3 votes
1 answer
673 views

Should reinforcement learning always assume (PO)MDP?

I recently just started learning reinforcement learning and learned that reinforcement learning algorithms work under the assumption of MDP or POMDP. However as I read A3C and recent vision based deep ...
haruishi's user avatar
1 vote
2 answers
64 views

Equations in "Intoduction to RL": What is the meaning and difference between E, and E with subscript?

This question is from An introduction to RL, page 78. In the formula below the page, both $\mathbb{E}$ and $\mathbb{E_\pi}$ are mentioned. Could you help me understand the difference between ...
Melanie A's user avatar
  • 167
1 vote
1 answer
283 views

MDP - RL, Multiple rewards for the same state possible?

This question is from An introduction to RL Pages 48 and 49. This question may also be related to below question, although I am not sure: Cannot see what the "notation abuse" is, mentioned ...
Melanie A's user avatar
  • 167
2 votes
1 answer
49 views

Using Policy Iteration on an automaton

I've read many explanation on how do to policy iteration, but I can't find an example, so I'm stuck right now trying to figure out to Policy Iteration. The numbers next to each state show the reward ...
Bar's user avatar
  • 23
3 votes
1 answer
28 views

What could be some Classification techniques to classify a tree of webpages given the category of each webpage

I want to perform a website classification task where I have modeled a website as a tree of webpages. I already have a model which can assign categories to the nodes in the tree (webpages). I need ...
Samyak Jain's user avatar
1 vote
1 answer
72 views

How to set the parameters of a Hidden Markov Model that'll be used to correct the mistakes by a previous classifier?

Say we've previously used a neural network or some other classifier C with $N$ training samples $I:=\{I_1,...I_N\}$ (that has a sequence or context, but is ignored by C) the, belonging to $K$ classes. ...
Sus_Q's user avatar
  • 11
3 votes
1 answer
154 views

What is the optimal value of a Markov Decision process with Single actions at each state?

I am trying to solve some questions about a MRP (i.e. a Markov Decision process with only one possible action at each state). The setup is as follows: There are two states ($a$ and $b$) stepping to $...
BlagBlug1987's user avatar
0 votes
2 answers
91 views

Simple Markov Chains Memoryless Property Question

I have a sequential data from time T1 to T6. The rows contain the sequence of states for 50 customers. There are only 3 states ...
mlgal55's user avatar
  • 43
4 votes
3 answers
5k views

Markov Chains for sequential data

I am new to Markov chains and HMM and I am looking for help in developing a program (in python) that predicts the next state based on 20 previous states (lets say 20 states in last 20 months). I have ...
mlgal55's user avatar
  • 43
1 vote
0 answers
23 views

Create a graphical viz of list of elements residing in a column in desired order [closed]

I have a list of elements in a column. Example: UID.................Flow 1............................qwerty, asdfgh, zxcvbn, poiuyt, lkjhgf, mnbvcx 2............................qpwoei, alskdj, ...
anirudh's user avatar
  • 11
2 votes
1 answer
346 views

How to use HMMs for continuous value prediction

I have some time-series data, which I need to use to predict a continuous value for a given time-stamp. I was initially doing it using a Multivariate Regression Model but I later figured that a time-...
Jeris's user avatar
  • 121
4 votes
1 answer
1k views

Viterbi-like algorithm suggesting top-N probable state sequences implementation

Traditional Viterbi algorithm (say, for hidden Markov models) provides the most probable hidden state sequence given a sequence of observations. There probably is an algorithm for decoding top-N ...
Anton's user avatar
  • 243