# Questions tagged [markov-process]

A Markov process is a stochastic process for which the Markov property holds: If you know the current state, then the next state is independent of all past states.

55 questions
Filter by
Sorted by
Tagged with
13k views

### Python library to implement Hidden Markov Models

What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before. Alternatively, is there a more ...
997 views

### Visualization of multiple Markov models

I am working on a project where we compare over 10 different Markov models, each representing a different treatment plan. Most often single models are visualized with a decision tree or transition ...
739 views

### Reward dependent on (state, action) versus (state, action, successor state)

I am studying reinforcement learning and I am working methodically through Sutton and Barto's book plus David Silver's lectures. I have noticed a minor difference in how the Markov Decision Processes ...
566 views

### How do I choose a discount factor in Markov Decision Problems?

I'm referring to the gamma in the Value function:
791 views

### Comparing transition matrices for Markov chains

I have a population, each unit of which exists in one of several states that change over time. I am using first-order Markov chains to model these state transitions. My population can be segmented ...
4k views

### Markov Chains for sequential data

I am new to Markov chains and HMM and I am looking for help in developing a program (in python) that predicts the next state based on 20 previous states (lets say 20 states in last 20 months). I have ...
637 views

### Viterbi-like algorithm suggesting top-N probable state sequences implementation

Traditional Viterbi algorithm (say, for hidden Markov models) provides the most probable hidden state sequence given a sequence of observations. There probably is an algorithm for decoding top-N ...
3k views

### Data visualization of frequencies of state transitions (possibly in R?)

I am working on some experimental data, which can be of types A, B and C. Now I observe this data for 5 time points, and I can see them move between A to B, B to C,... etc. I see such transitions for ...
136 views

### When to stop calculating values of each cell in the grid in Reinforcement Learning(dynamic programming) applied on gridworld

Considering application of Reinforcement learning(dynamic programming method performing value iteration) on grid world, in each of the iteration, I go through each of the cell of the grid and update ...
120 views

### Using dhmm_em to form the hmm of mfccs' from song clips

I was working on a project on music genre classification and decided to use an hmm to model my data. After extracting the mfccs' from multiple song clips I get multiple matrices of size 13 * x (...
164 views

### Markov Chains: How much steps to conclude a Transition Matrix

I have just learned Markov Chains which I am using to model a real world problem. The model comprises 3 states [a b c]. For now I am collection data and calculating ...
23 views

### What could be some Classification techniques to classify a tree of webpages given the category of each webpage

I want to perform a website classification task where I have modeled a website as a tree of webpages. I already have a model which can assign categories to the nodes in the tree (webpages). I need ...
67 views

24 views

### If I use Gibbs sampling with a Bayesian model, what do I have to check is memoryless?

Right now I am trying to better understand how Bayesian modeling works with just the basics. I found through reading tutorials that some very basic Bayesian models like Bayesian Hierarchical Modeling ...
198 views

### How to use HMMs for continuous value prediction

I have some time-series data, which I need to use to predict a continuous value for a given time-stamp. I was initially doing it using a Multivariate Regression Model but I later figured that a time-...
47 views

### Equations in “Intoduction to RL”: What is the meaning and difference between E, and E with subscript?

This question is from An introduction to RL, page 78. In the formula below the page, both $\mathbb{E}$ and $\mathbb{E_\pi}$ are mentioned. Could you help me understand the difference between ...
80 views

### Q-learning when minimising a total cost instead of maximising a total reward

I have a decision problem where the results are measured as a cost that I want to minimise. It seems like a good fit to Q-learning, but I am not sure how to adjust it to deal with a cost instead of a ...
100 views

### Predict how many days late or early someone will finish their work

So I have a set of deadlines and people, with a database of when those people finished their previous work and how much after the deadline it was, as well as when the work was given. The work itself ...
48 views

### Evaluating value functions in RL

I'm working my way through the book Reinforcement Learning by Richar S. Sutton and Andrew G. Barto and I am stuck on the following question. The value of a state depends on the the values of the ...
168 views

### Reinforcement Learning - How are these state values in MRP calculated?

This is a question from the book an Introduction to RL, page 125, example 6.2. The example compares the prediction abilities of TD(0) and constant $\alpha$ MC when applied to the below Markov ...
230 views

### Should reinforcement learning always assume (PO)MDP?

I recently just started learning reinforcement learning and learned that reinforcement learning algorithms work under the assumption of MDP or POMDP. However as I read A3C and recent vision based deep ...
102 views

### MDP - RL, Multiple rewards for the same state possible?

This question is from An introduction to RL Pages 48 and 49. This question may also be related to below question, although I am not sure: Cannot see what the "notation abuse" is, mentioned ...
4k views

### finding themes from text documents

I have a text documents that contain 1000s of abstracts from medical whitepapers. I want to find themes from that text. Any suggestions other than text clustering since clustering helped me to find ...
242 views

### How to estimate the transition probabilities for a Markov Chain when time intervals are non-equally spaced

I've been given a dataset with a number of observable states. I am trying to apply a Finite State Markov Chain to model the system, but I found that I can't estimate the transition probabilities if ...
27 views

### Regime detection to identify transitions between habitats

The following figure represents the concentration of a substance (referred to as Element in the code) measured in an organism throughout its life. There are ...
11 views

7 views

### Hidden Markov Model with Autoregressive emission model?

So far, all standard HMM implementations I've seen assume some variation of a Gaussian Mixture (GMM) as their emission model. It can of course only have a single mixture component which reduces it to ...
25 views

### Applying Reinforcement Learning in the following scenario

I'm working on a scenario/environment where I have a simulation that provides an arrangement or results of the simulation that has data in a format of samples in vectors(x,y,z,N). Let's say it maps ...
19 views

### Find changes in variables into two states

I have a dataframe like this: ...