Questions tagged [markov-process]

A Markov process is a stochastic process for which the Markov property holds: If you know the current state, then the next state is independent of all past states.

29 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6 votes
0 answers
1k views

Comparing transition matrices for Markov chains

I have a population, each unit of which exists in one of several states that change over time. I am using first-order Markov chains to model these state transitions. My population can be segmented ...
user avatar
  • 1,037
3 votes
1 answer
43 views

If I use Gibbs sampling with a Bayesian model, what do I have to check is memoryless?

Right now I am trying to better understand how Bayesian modeling works with just the basics. I found through reading tutorials that some very basic Bayesian models like Bayesian Hierarchical Modeling ...
user avatar
3 votes
0 answers
64 views

Identifying sequences of behavioral interactions between multiple individuals

I'm wondering if anyone might have some novel insights as to the best way to analyze the following data. It's a problem I've been thinking about in the back of my mind for a while, so I thought that ...
user avatar
  • 131
2 votes
0 answers
27 views

"Learning" algorithm to use when future depends on past events (MDP property not met)

There are around 5 different retirement plans available in my country. People can pick from them freely. I would like to create a solution that would try to predict the best plan(s) given a particular ...
user avatar
2 votes
1 answer
247 views

How to use HMMs for continuous value prediction

I have some time-series data, which I need to use to predict a continuous value for a given time-stamp. I was initially doing it using a Multivariate Regression Model but I later figured that a time-...
user avatar
  • 121
1 vote
1 answer
10 views

memory error- python N-th order Markovian transition matrix from a given sequence

Ok. What is wrong with you code! I am trying to calculate transition probabilities for each leg. The code works for small array but for the actual dataset I got memory error. I have 64 g version ...
user avatar
1 vote
0 answers
20 views

Jacks car rental problem: why deterministic policies?

In Sutton & Barto Book: Reinforcement Learning: An Introduction, there is the following problem: I have this question: why are the policies to be considered here are deterministic?
user avatar
1 vote
0 answers
13 views

global independence vs local independence in markov network

I could not understand the local independence and global independence of a markov network. Please help me understand with a simple graph
user avatar
  • 135
1 vote
0 answers
32 views

state transition classification on terminal state

I have data on a unit $i$ which enters an entry state $S_0$. This unit has some covariates $x_i$ I would like to predict the probability the unit will reach the terminal state $S_{pos}$ or $S_{neg}$. ...
user avatar
1 vote
0 answers
39 views

How do I test and validate a Markov Model in RStudio?

I've created a Markov Model in RStudio using the seqHMM library. I'm new to R and would usually do this kind of thing in Python, but I wanted to use seqHMM. I can see the model has been created and it ...
user avatar
1 vote
0 answers
60 views

Ranking graph's nodes by score propagation

Problem I have the following directed tripartite graph $G(E\cup V\cup P, A)$, where there is a many-to-one symmetric relationship between the subsets V and E - $e\in E,v\in V,[e, v]\in A \iff [v, e]\...
user avatar
  • 111
1 vote
0 answers
15 views

Hidden Markov Model with Autoregressive emission model?

So far, all standard HMM implementations I've seen assume some variation of a Gaussian Mixture (GMM) as their emission model. It can of course only have a single mixture component which reduces it to ...
user avatar
1 vote
0 answers
25 views

Estimating model for transition probabilities of a Markov Chain

Suppose that I have a Markov chain with $S$ states evolving over time. I have $S^2\times T$ values of the transition matrix, where $T$ is the number of time periods. I also have $K$ matrices $X$ of $T\...
user avatar
1 vote
2 answers
93 views

Machine Learning algorithm for detecting anomalies in large sets of events

Let's start with the following hypothetical preconditions: There is traffic: normal and anomaly. Each traffic sample contains a list of events (of variable size) Events happen in order, the possible ...
user avatar
1 vote
0 answers
57 views

Reinforcement learning - generating a matrix of continuous values with varying size for test data generation

Currently, I am using RL A3C algorithm for test data generation, where for a set of 30 functions written in C (mostly basic algorithms like Prime number checks, triangle validity, etc.) I try to ...
user avatar
  • 21
1 vote
1 answer
64 views

How to set the parameters of a Hidden Markov Model that'll be used to correct the mistakes by a previous classifier?

Say we've previously used a neural network or some other classifier C with $N$ training samples $I:=\{I_1,...I_N\}$ (that has a sequence or context, but is ignored by C) the, belonging to $K$ classes. ...
user avatar
  • 11
1 vote
0 answers
1k views

Markov chain modelling?

I am working on a personal loan dataset. For each loan, we recorded its credit status monthly after it was drawn by the borrower. Let's say there were 6 status coded by A-F. My project is to use ...
user avatar
  • 753
1 vote
0 answers
277 views

Online methods for sequence prediction

A ml beginner here, so please bear with me. If I understand correctly RNNs seem to be the go to method right now for sequence prediction for a given input (single/as a sequence). But I do not have ...
user avatar
0 votes
0 answers
5 views

How to deal with multiple possible rewards in a transition in MDP

Suppose there is a state S with two transitions under action A but both transited states are ...
user avatar
  • 1
0 votes
0 answers
34 views

Using Markov Chains in digital customer journey

I have data of each page visited by a customer in a session, my objective is to find out the most optimal path where we see the maximum conversion rate. My idea Is to use Markov Chain to identify that ...
user avatar
0 votes
0 answers
12 views

Compute for policy the state value function v(s) for each state

The instruction of the question: State A is absorbing. Transition to A from state 1 or 4 yields an immediate reward of 12. All other transitions incur a reward of 1. Transitions are deterministic (i.e....
user avatar
0 votes
0 answers
12 views

Method for predicting future state, based on time spent in previous states

So what I'm looking for is the best approach to predict a future state. Say we have three states: A, B, C. I want to predict if in the next time-interval (f.e. a day or a week) the state will become C....
user avatar
  • 146
0 votes
0 answers
11 views

Value function when the policy is deterministic

This is the value function expression for a stochastic policy: $\displaystyle v_{\pi}(s)=\sum_{a \in \mathcal{A}}\pi(a|s)\bigg(\mathcal{R}_s^a+\gamma \sum_{s' \in \mathcal{S}} \mathbb{P}_{ss'}^a v_{\...
user avatar
0 votes
0 answers
9 views

Computing the state-value function of a Markov decision process from the classical definition

For the above Markov decision process under given action policy $a_1$, how can I determine the value of state $s_1$ using the state-value definition $v(s)=E[G_t| S_t=s]$ where $G_t$ is the return? ...
user avatar
0 votes
1 answer
32 views

Find changes in variables into two states

I have a dataframe like this: ...
user avatar
  • 147
0 votes
0 answers
22 views

Machine Learning alternative for hashing

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
user avatar
0 votes
0 answers
163 views

Best python library for training using Hidden Marov model with Gaussian Mixture

I would like to train my data using HMM- GMM (Baum Welch approach with gaussian Mixture) to find the best parameters suited for my data. Note : My data is ...
user avatar
  • 155
0 votes
2 answers
81 views

Simple Markov Chains Memoryless Property Question

I have a sequential data from time T1 to T6. The rows contain the sequence of states for 50 customers. There are only 3 states ...
user avatar
  • 43
0 votes
0 answers
525 views

HMM - Matlab for data set to detect anomaly

I have a dataset of oil temperatures. The time series consist of 100 hours of measurement at every second. There is an anomaly in the data that I would like to detect using Hidden Markov Models (HMM). ...
user avatar
  • 9