# Questions tagged [markov-process]

A Markov process is a stochastic process for which the Markov property holds: If you know the current state, then the next state is independent of all past states.

29 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1k views

### Comparing transition matrices for Markov chains

I have a population, each unit of which exists in one of several states that change over time. I am using first-order Markov chains to model these state transitions. My population can be segmented ...
• 1,037
43 views

### If I use Gibbs sampling with a Bayesian model, what do I have to check is memoryless?

Right now I am trying to better understand how Bayesian modeling works with just the basics. I found through reading tutorials that some very basic Bayesian models like Bayesian Hierarchical Modeling ...
64 views

### Identifying sequences of behavioral interactions between multiple individuals

I'm wondering if anyone might have some novel insights as to the best way to analyze the following data. It's a problem I've been thinking about in the back of my mind for a while, so I thought that ...
• 131
27 views

### "Learning" algorithm to use when future depends on past events (MDP property not met)

There are around 5 different retirement plans available in my country. People can pick from them freely. I would like to create a solution that would try to predict the best plan(s) given a particular ...
• 121
247 views

### How to use HMMs for continuous value prediction

I have some time-series data, which I need to use to predict a continuous value for a given time-stamp. I was initially doing it using a Multivariate Regression Model but I later figured that a time-...
• 121
1 vote
10 views

### memory error- python N-th order Markovian transition matrix from a given sequence

Ok. What is wrong with you code! I am trying to calculate transition probabilities for each leg. The code works for small array but for the actual dataset I got memory error. I have 64 g version ...
1 vote
20 views

### Jacks car rental problem: why deterministic policies?

In Sutton & Barto Book: Reinforcement Learning: An Introduction, there is the following problem: I have this question: why are the policies to be considered here are deterministic?
1 vote
13 views

### global independence vs local independence in markov network

I could not understand the local independence and global independence of a markov network. Please help me understand with a simple graph
• 135
1 vote
32 views

### state transition classification on terminal state

I have data on a unit $i$ which enters an entry state $S_0$. This unit has some covariates $x_i$ I would like to predict the probability the unit will reach the terminal state $S_{pos}$ or $S_{neg}$. ...
1 vote
39 views

### How do I test and validate a Markov Model in RStudio?

I've created a Markov Model in RStudio using the seqHMM library. I'm new to R and would usually do this kind of thing in Python, but I wanted to use seqHMM. I can see the model has been created and it ...
• 111
1 vote
60 views

1 vote
93 views

### Machine Learning algorithm for detecting anomalies in large sets of events

Let's start with the following hypothetical preconditions: There is traffic: normal and anomaly. Each traffic sample contains a list of events (of variable size) Events happen in order, the possible ...
1 vote
57 views

### Reinforcement learning - generating a matrix of continuous values with varying size for test data generation

Currently, I am using RL A3C algorithm for test data generation, where for a set of 30 functions written in C (mostly basic algorithms like Prime number checks, triangle validity, etc.) I try to ...
• 21
1 vote
64 views

### How to set the parameters of a Hidden Markov Model that'll be used to correct the mistakes by a previous classifier?

Say we've previously used a neural network or some other classifier C with $N$ training samples $I:=\{I_1,...I_N\}$ (that has a sequence or context, but is ignored by C) the, belonging to $K$ classes. ...
• 11
1 vote
1k views

### Markov chain modelling?

I am working on a personal loan dataset. For each loan, we recorded its credit status monthly after it was drawn by the borrower. Let's say there were 6 status coded by A-F. My project is to use ...
• 753
1 vote
277 views

### Online methods for sequence prediction

A ml beginner here, so please bear with me. If I understand correctly RNNs seem to be the go to method right now for sequence prediction for a given input (single/as a sequence). But I do not have ...
5 views

### How to deal with multiple possible rewards in a transition in MDP

Suppose there is a state S with two transitions under action A but both transited states are ...
34 views

### Using Markov Chains in digital customer journey

I have data of each page visited by a customer in a session, my objective is to find out the most optimal path where we see the maximum conversion rate. My idea Is to use Markov Chain to identify that ...
12 views

### Compute for policy the state value function v(s) for each state

The instruction of the question: State A is absorbing. Transition to A from state 1 or 4 yields an immediate reward of 12. All other transitions incur a reward of 1. Transitions are deterministic (i.e....
12 views

### Method for predicting future state, based on time spent in previous states

So what I'm looking for is the best approach to predict a future state. Say we have three states: A, B, C. I want to predict if in the next time-interval (f.e. a day or a week) the state will become C....
• 146
11 views

32 views

### Find changes in variables into two states

I have a dataframe like this: ...
• 147
22 views

### Machine Learning alternative for hashing

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
• 101
163 views

### Best python library for training using Hidden Marov model with Gaussian Mixture

I would like to train my data using HMM- GMM (Baum Welch approach with gaussian Mixture) to find the best parameters suited for my data. Note : My data is ...
• 155