# Questions tagged [markov-process]

A Markov process is a stochastic process for which the Markov property holds: If you know the current state, then the next state is independent of all past states.

62 questions
Filter by
Sorted by
Tagged with
15k views

### Python library to implement Hidden Markov Models

What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before. Alternatively, is there a more ...
2k views

### Visualization of multiple Markov models

I am working on a project where we compare over 10 different Markov models, each representing a different treatment plan. Most often single models are visualized with a decision tree or transition ...
851 views

### How do I choose a discount factor in Markov Decision Problems?

I'm referring to the gamma in the Value function:
915 views

### Reward dependent on (state, action) versus (state, action, successor state)

I am studying reinforcement learning and I am working methodically through Sutton and Barto's book plus David Silver's lectures. I have noticed a minor difference in how the Markov Decision Processes ...
1k views

### Comparing transition matrices for Markov chains

I have a population, each unit of which exists in one of several states that change over time. I am using first-order Markov chains to model these state transitions. My population can be segmented ...
4k views

### Markov Chains for sequential data

I am new to Markov chains and HMM and I am looking for help in developing a program (in python) that predicts the next state based on 20 previous states (lets say 20 states in last 20 months). I have ...
7k views

### Markov switching models

What are some reference sources for understanding Markov switching models?
176 views

### Markov Chains: How much steps to conclude a Transition Matrix

I have just learned Markov Chains which I am using to model a real world problem. The model comprises 3 states [a b c]. For now I am collection data and calculating ...
30 views

### Artificially increasing frequency weight of word ending characters in word building

I have a database of letter pair bigrams. For example: ...
864 views

### Viterbi-like algorithm suggesting top-N probable state sequences implementation

Traditional Viterbi algorithm (say, for hidden Markov models) provides the most probable hidden state sequence given a sequence of observations. There probably is an algorithm for decoding top-N ...
7k views

### What is the relationship between MDP and RL?

What is the relationship between Markov Decision Processes and Reinforcement Learning? Could we say RL and DP are two types of MDP?
3k views

### Data visualization of frequencies of state transitions (possibly in R?)

I am working on some experimental data, which can be of types A, B and C. Now I observe this data for 5 time points, and I can see them move between A to B, B to C,... etc. I see such transitions for ...
145 views

### When to stop calculating values of each cell in the grid in Reinforcement Learning(dynamic programming) applied on gridworld

Considering application of Reinforcement learning(dynamic programming method performing value iteration) on grid world, in each of the iteration, I go through each of the cell of the grid and update ...
419 views

### Should reinforcement learning always assume (PO)MDP?

I recently just started learning reinforcement learning and learned that reinforcement learning algorithms work under the assumption of MDP or POMDP. However as I read A3C and recent vision based deep ...
129 views

### Using dhmm_em to form the hmm of mfccs' from song clips

I was working on a project on music genre classification and decided to use an hmm to model my data. After extracting the mfccs' from multiple song clips I get multiple matrices of size 13 * x (...
25 views

### What could be some Classification techniques to classify a tree of webpages given the category of each webpage

I want to perform a website classification task where I have modeled a website as a tree of webpages. I already have a model which can assign categories to the nodes in the tree (webpages). I need ...
98 views

44 views

### Using Policy Iteration on an automaton

I've read many explanation on how do to policy iteration, but I can't find an example, so I'm stuck right now trying to figure out to Policy Iteration. The numbers next to each state show the reward ...
264 views

### Can I apply Hidden Markov Models this way?

I have just gotten my feet wet with Hidden Markov Models. Now I want to apply them to tell whether a transaction from an ATM is suspicious or not. I have great confusion in defining my Hidden States. ...
26 views

### "Learning" algorithm to use when future depends on past events (MDP property not met)

There are around 5 different retirement plans available in my country. People can pick from them freely. I would like to create a solution that would try to predict the best plan(s) given a particular ...
40 views

### Regime detection to identify transitions between habitats

The following figure represents the concentration of a substance (referred to as Element in the code) measured in an organism throughout its life. There are ...
220 views

### How to use HMMs for continuous value prediction

I have some time-series data, which I need to use to predict a continuous value for a given time-stamp. I was initially doing it using a Multivariate Regression Model but I later figured that a time-...
55 views

### Equations in "Intoduction to RL": What is the meaning and difference between E, and E with subscript?

This question is from An introduction to RL, page 78. In the formula below the page, both $\mathbb{E}$ and $\mathbb{E_\pi}$ are mentioned. Could you help me understand the difference between ...
434 views

### Q-learning when minimising a total cost instead of maximising a total reward

I have a decision problem where the results are measured as a cost that I want to minimise. It seems like a good fit to Q-learning, but I am not sure how to adjust it to deal with a cost instead of a ...
6k views

### finding themes from text documents

I have a text documents that contain 1000s of abstracts from medical whitepapers. I want to find themes from that text. Any suggestions other than text clustering since clustering helped me to find ...
38 views

### Markov Chain vs Bayes Net

I am learning about Markov Chain and Bayesian Nets. However at this point I am a bit confused about what types of problems are modelled with the two different models presented to us. From what I ...
69 views

### Choosing a model for input: categorised, weighted sequence, output: binary variable

What would be an appropriate model for predicting a binary target variable, given a weighted sequence? Sequences will be reasonably short, typically between ~ 1 and 5 elements. Illustrated example Say ...
210 views

### Predict how many days late or early someone will finish their work

So I have a set of deadlines and people, with a database of when those people finished their previous work and how much after the deadline it was, as well as when the work was given. The work itself ...
82 views

### Evaluating value functions in RL

I'm working my way through the book Reinforcement Learning by Richar S. Sutton and Andrew G. Barto and I am stuck on the following question. The value of a state depends on the the values of the ...
153 views

### MDP - RL, Multiple rewards for the same state possible?

This question is from An introduction to RL Pages 48 and 49. This question may also be related to below question, although I am not sure: Cannot see what the "notation abuse" is, mentioned ...
259 views

### How to estimate the transition probabilities for a Markov Chain when time intervals are non-equally spaced

I've been given a dataset with a number of observable states. I am trying to apply a Finite State Markov Chain to model the system, but I found that I can't estimate the transition probabilities if ...
13 views

### global independence vs local independence in markov network

I could not understand the local independence and global independence of a markov network. Please help me understand with a simple graph
25 views

### state transition classification on terminal state

I have data on a unit $i$ which enters an entry state $S_0$. This unit has some covariates $x_i$ I would like to predict the probability the unit will reach the terminal state $S_{pos}$ or $S_{neg}$. ...
33 views

### How do I test and validate a Markov Model in RStudio?

I've created a Markov Model in RStudio using the seqHMM library. I'm new to R and would usually do this kind of thing in Python, but I wanted to use seqHMM. I can see the model has been created and it ...
45 views

88 views

### Machine Learning algorithm for detecting anomalies in large sets of events

Let's start with the following hypothetical preconditions: There is traffic: normal and anomaly. Each traffic sample contains a list of events (of variable size) Events happen in order, the possible ...
45 views

### Reinforcement learning - generating a matrix of continuous values with varying size for test data generation

Currently, I am using RL A3C algorithm for test data generation, where for a set of 30 functions written in C (mostly basic algorithms like Prime number checks, triangle validity, etc.) I try to ...
58 views

### How to set the parameters of a Hidden Markov Model that'll be used to correct the mistakes by a previous classifier?

Say we've previously used a neural network or some other classifier C with $N$ training samples $I:=\{I_1,...I_N\}$ (that has a sequence or context, but is ignored by C) the, belonging to $K$ classes. ...
20 views

### Create a graphical viz of list of elements residing in a column in desired order [closed]

I have a list of elements in a column. Example: UID.................Flow 1............................qwerty, asdfgh, zxcvbn, poiuyt, lkjhgf, mnbvcx 2............................qpwoei, alskdj, ...