Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-process]

A Markov process is a stochastic process for which the Markov property holds: If you know the current state, then the next state is independent of all past states.

0
votes
0answers
14 views

How apply Reinforcement Learning in the following case?

Suppose that I have to move from point A to B and I have to choose among 3 different paths. But we don't know the traffic in each path, so what is the training rule to use to learn the best behaviour? ...
0
votes
0answers
4 views

Probabilistic search engine user behavior markov models

I am considering the following probabilistic Markov model of actions of a user on the results page of a search engine. The user examines the first result, with a probability $A$ he is satisfied with ...
0
votes
0answers
16 views

Which one to maximise? Q-value or V-value?

I just start learning reinforcement learning and studying Markov Decision Process (MDP). The state-value function (V-value, $v_{\pi}(s)$) and action-value function (Q-value, $q_{\pi}(s, a)$) are \...
0
votes
0answers
10 views

Code for Point Process Inference

I need to perform inference on a point process, by which I mean I need a statistical model from which I can (at least) estimate likelihoods. The underlying process appears to have a latent structure, ...
1
vote
1answer
39 views

Reinforcement Learning - How are these state values in MRP calculated?

This is a question from the book an Introduction to RL, page 125, example 6.2. The example compares the prediction abilities of TD(0) and constant $ \alpha $ MC when applied to the below Markov ...
1
vote
0answers
21 views

have deep reinforcement learning algorithms the unified behavior?

Could we say the behavior of different deep reinforcement learning algorithms is very similar for MDP and POMDP? Could we say DRL algorithms present a unified approach for finite-horizon, infinite ...
2
votes
1answer
51 views

What are the differences between Reinforcement Learning (RL) and Supervised Learning?

What is the difference between Reinforcement Learning (RL) and Supervised Learning? Does RL hava more difficulty in finding a stable solution? Does Q-learning have more difficulty in finding a ...
0
votes
1answer
287 views

What is the relationship between MDP and RL?

What is the relationship between Markov Decision Processes and Reinforcement Learning? Could we say RL and DP are two types of MDP?
1
vote
1answer
58 views

Should reinforcement learning always assume (PO)MDP?

I recently just started learning reinforcement learning and learned that reinforcement learning algorithms work under the assumption of MDP or POMDP. However as I read A3C and recent vision based deep ...
1
vote
2answers
39 views

Equations in “Intoduction to RL”: What is the meaning and difference between E, and E with subscript?

This question is from An introduction to RL, page 78. In the formula below the page, both $\mathbb{E}$ and $\mathbb{E_\pi}$ are mentioned. Could you help me understand the difference between ...
1
vote
1answer
31 views

MDP - RL, Multiple rewards for the same state possible?

This question is from An introduction to RL Pages 48 and 49. This question may also be related to below question, although I am not sure: Cannot see what the "notation abuse" is, mentioned ...
2
votes
1answer
33 views

Using Policy Iteration on an automaton

I've read many explanation on how do to policy iteration, but I can't find an example, so I'm stuck right now trying to figure out to Policy Iteration. The numbers next to each state show the reward ...
3
votes
1answer
22 views

What could be some Classification techniques to classify a tree of webpages given the category of each webpage

I want to perform a website classification task where I have modeled a website as a tree of webpages. I already have a model which can assign categories to the nodes in the tree (webpages). I need ...
0
votes
1answer
37 views

How to set the parameters of a Hidden Markov Model that'll be used to correct the mistakes by a previous classifier?

Say we've previously used a neural network or some other classifier C with $N$ training samples $I:=\{I_1,...I_N\}$ (that has a sequence or context, but is ignored by C) the, belonging to $K$ classes. ...
0
votes
0answers
18 views

How to cross-validate a markov model

What is the process for validating the Markov model? Suppose I have 38 files which contain sequences(i.e A, B, B, C, E, D, E......). all the files contain different numbers of sequence (i.e. 1st file ...
3
votes
1answer
53 views

What is the optimal value of a Markov Decision process with Single actions at each state?

I am trying to solve some questions about a MRP (i.e. a Markov Decision process with only one possible action at each state). The setup is as follows: There are two states ($a$ and $b$) stepping to $...
1
vote
2answers
45 views

Simple Markov Chains Memoryless Property Question

I have a sequential data from time T1 to T6. The rows contain the sequence of states for 50 customers. There are only 3 states ...
3
votes
3answers
1k views

Markov Chains for sequential data

I am new to Markov chains and HMM and I am looking for help in developing a program (in python) that predicts the next state based on 20 previous states (lets say 20 states in last 20 months). I have ...
0
votes
0answers
47 views

generating training samples for an RNN

I am trying to learn how an recurrent neural network (RNN) is trained. I have read chapter 10 of the book by Goodfellow, Bengio and Courville and various tutorials on the internet. If I understand ...
1
vote
0answers
12 views

Create a graphical viz of list of elements residing in a column in desired order [closed]

I have a list of elements in a column. Example: UID.................Flow 1............................qwerty, asdfgh, zxcvbn, poiuyt, lkjhgf, mnbvcx 2............................qpwoei, alskdj, ...
2
votes
0answers
142 views

How to use HMMs for continuous value prediction

I have some time-series data, which I need to use to predict a continuous value for a given time-stamp. I was initially doing it using a Multivariate Regression Model but I later figured that a time-...
4
votes
1answer
213 views

Viterbi-like algorithm suggesting top-N probable state sequences implementation

Traditional Viterbi algorithm (say, for hidden Markov models) provides the most probable hidden state sequence given a sequence of observations. There probably is an algorithm for decoding top-N ...
0
votes
1answer
2k views

finding themes from text documents

I have a text documents that contain 1000s of abstracts from medical whitepapers. I want to find themes from that text. Any suggestions other than text clustering since clustering helped me to find ...
1
vote
1answer
162 views

How to estimate the transition probabilities for a Markov Chain when time intervals are non-equally spaced

I've been given a dataset with a number of observable states. I am trying to apply a Finite State Markov Chain to model the system, but I found that I can't estimate the transition probabilities if ...
3
votes
0answers
407 views

Comparing transition matrices for Markov chains

I have a population, each unit of which exists in one of several states that change over time. I am using first-order Markov chains to model these state transitions. My population can be segmented ...
1
vote
0answers
44 views

Suggestions the way to start [closed]

One of our famous mathematicians, James Simons, used an extension of the Baum-Welch algorithm to 'crack' the wall street when he started trading on the stock market. Now, as Google, all informations ...
5
votes
1answer
351 views

Reward dependent on (state, action) versus (state, action, successor state)

I am studying reinforcement learning and I am working methodically through Sutton and Barto's book plus David Silver's lectures. I have noticed a minor difference in how the Markov Decision Processes ...
1
vote
0answers
400 views

HMM - Matlab for data set to detect anomaly

I have a dataset of oil temperatures. The time series consist of 100 hours of measurement at every second. There is an anomaly in the data that I would like to detect using Hidden Markov Models (HMM). ...
1
vote
0answers
899 views

Markov chain modelling?

I am working on a personal loan dataset. For each loan, we recorded its credit status monthly after it was drawn by the borrower. Let's say there were 6 status coded by A-F. My project is to use ...
7
votes
0answers
557 views

Visualization of multiple Markov models

I am working on a project where we compare over 10 different Markov models, each representing a different treatment plan. Most often single models are visualized with a decision tree or transition ...
3
votes
1answer
97 views

Using dhmm_em to form the hmm of mfccs' from song clips

I was working on a project on music genre classification and decided to use an hmm to model my data. After extracting the mfccs' from multiple song clips I get multiple matrices of size 13 * x (...
1
vote
0answers
181 views

Online methods for sequence prediction

A ml beginner here, so please bear with me. If I understand correctly RNNs seem to be the go to method right now for sequence prediction for a given input (single/as a sequence). But I do not have ...
3
votes
3answers
2k views

Data visualization of frequencies of state transitions (possibly in R?)

I am working on some experimental data, which can be of types A, B and C. Now I observe this data for 5 time points, and I can see them move between A to B, B to C,... etc. I see such transitions for ...
1
vote
1answer
4k views

Markov switching models

What are some reference sources for understanding Markov switching models?
6
votes
2answers
319 views

How do I choose a discount factor in Markov Decision Problems?

I'm referring to the gamma in the Value function:
2
votes
1answer
196 views

Can I apply Hidden Markov Models this way?

I have just gotten my feet wet with Hidden Markov Models. Now I want to apply them to tell whether a transaction from an ATM is suspicious or not. I have great confusion in defining my Hidden States. ...
0
votes
2answers
2k views

Can HMM be used as a binary classifier?

I have some time-series data, which I need to use to predict a binary label for a given time-stamp. I was initially doing it using a Logistic Regression Model but I later figured that a time-series ...
14
votes
5answers
9k views

Python library to implement Hidden Markov Models

What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before. Alternatively, is there a more ...
1
vote
1answer
80 views

How can I rank paths through an HMM?

I have a profile hidden Markov model that I use to identify all instances of a user-defined pattern of symbols in a long sequence of symbols. I use the Viterbi algorithm to find the most probable path ...
2
votes
1answer
1k views

emission probability using hmmlearn package in python

I am learning hmm and try to implement it in Python hmmlearn package(http://hmmlearn.github.io/hmmlearn/hmm.html#building-hmm-and-generating-samples). However I am not quite understand what the ...
3
votes
1answer
120 views

When to stop calculating values of each cell in the grid in Reinforcement Learning(dynamic programming) applied on gridworld

Considering application of Reinforcement learning(dynamic programming method performing value iteration) on grid world, in each of the iteration, I go through each of the cell of the grid and update ...
3
votes
1answer
151 views

Markov Chains: How much steps to conclude a Transition Matrix

I have just learned Markov Chains which I am using to model a real world problem. The model comprises 3 states [a b c]. For now I am collection data and calculating ...
3
votes
0answers
52 views

Identifying sequences of behavioral interactions between multiple individuals

I'm wondering if anyone might have some novel insights as to the best way to analyze the following data. It's a problem I've been thinking about in the back of my mind for a while, so I thought that ...
0
votes
1answer
64 views

Matlab simulation through FIS and Markov Process

I need to simulate for an academical project how the traffic fluxes (input/output with respect to a monitored area, measured in number of cars) of a city area evolves in correspondence of an event (i....