Questions tagged [markov-process]

A Markov process is a stochastic process for which the Markov property holds: If you know the current state, then the next state is independent of all past states.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Should I update action value functions when there is no change?

Suppose there is a website and the decision-maker wants to recommend some products to each customer visiting the website. Customers visit the websites in which the time interval between two ...
0
votes
2answers
60 views

Machine Learning algorithm for detecting anomalies in large sets of events

Let's start with the following hypothetical preconditions: There is traffic: normal and anomaly. Each traffic sample contains a list of events (of variable size) Events happen in order, the possible ...
1
vote
1answer
27 views

Regime detection to identify transitions between habitats

The following figure represents the concentration of a substance (referred to as Element in the code) measured in an organism throughout its life. There are ...
0
votes
0answers
11 views

Ranking graph's nodes by score propagation

Problem I have the following directed tripartite graph $G(E\cup V\cup P, A)$, where there is a many-to-one symmetric relationship between the subsets V and E - $e\in E,v\in V,[e, v]\in A \iff [v, e]\...
0
votes
0answers
7 views

Hidden Markov Model with Autoregressive emission model?

So far, all standard HMM implementations I've seen assume some variation of a Gaussian Mixture (GMM) as their emission model. It can of course only have a single mixture component which reduces it to ...
0
votes
1answer
52 views

How to set the parameters of a Hidden Markov Model that'll be used to correct the mistakes by a previous classifier?

Say we've previously used a neural network or some other classifier C with $N$ training samples $I:=\{I_1,...I_N\}$ (that has a sequence or context, but is ignored by C) the, belonging to $K$ classes. ...
0
votes
1answer
24 views

Reinforcement Learning control with known dynamic equation

I know there is model-based reinforcement learning. But all the approaches assume an MDP. If I want to do a feedback control of a system (i. e. control an inverted pendulum) it's quite easy to find ...
16
votes
1answer
1k views

Visualization of multiple Markov models

I am working on a project where we compare over 10 different Markov models, each representing a different treatment plan. Most often single models are visualized with a decision tree or transition ...
0
votes
2answers
61 views

Simple Markov Chains Memoryless Property Question

I have a sequential data from time T1 to T6. The rows contain the sequence of states for 50 customers. There are only 3 states ...
1
vote
0answers
11 views

Estimating model for transition probabilities of a Markov Chain

Suppose that I have a Markov chain with $S$ states evolving over time. I have $S^2\times T$ values of the transition matrix, where $T$ is the number of time periods. I also have $K$ matrices $X$ of $T\...
2
votes
0answers
18 views

Markov Decision Process representation

I'm attempting to model a simple process using a Markov Decision Process. Let $A$ be a set of $3$ actions : $ A \in \{b,s\}$. $T(s,a,s')$ represents the probability of if in state $s$ , take action $...
3
votes
0answers
26 views

If I use Gibbs sampling with a Bayesian model, what do I have to check is memoryless?

Right now I am trying to better understand how Bayesian modeling works with just the basics. I found through reading tutorials that some very basic Bayesian models like Bayesian Hierarchical Modeling ...
0
votes
0answers
25 views

Applying Reinforcement Learning in the following scenario

I'm working on a scenario/environment where I have a simulation that provides an arrangement or results of the simulation that has data in a format of samples in vectors(x,y,z,N). Let's say it maps ...
0
votes
0answers
17 views

Labelling through Markov Switching models

I am new to machine learning, in the past I have worked with time series techniques. I have a database composed by the financial time series of SPX, its Exponential Moving average 20days, BBB ...
0
votes
0answers
19 views

Machine Learning alternative for hashing

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
0
votes
0answers
66 views

Best python library for training using Hidden Marov model with Gaussian Mixture

I would like to train my data using HMM- GMM (Baum Welch approach with gaussian Mixture) to find the best parameters suited for my data. Note : My data is ...
1
vote
1answer
82 views

Q-learning when minimising a total cost instead of maximising a total reward

I have a decision problem where the results are measured as a cost that I want to minimise. It seems like a good fit to Q-learning, but I am not sure how to adjust it to deal with a cost instead of a ...
1
vote
1answer
101 views

Predict how many days late or early someone will finish their work

So I have a set of deadlines and people, with a database of when those people finished their previous work and how much after the deadline it was, as well as when the work was given. The work itself ...
1
vote
0answers
14 views
1
vote
0answers
32 views

Reinforcement learning - generating a matrix of continuous values with varying size for test data generation

Currently, I am using RL A3C algorithm for test data generation, where for a set of 30 functions written in C (mostly basic algorithms like Prime number checks, triangle validity, etc.) I try to ...
1
vote
1answer
48 views

Evaluating value functions in RL

I'm working my way through the book Reinforcement Learning by Richar S. Sutton and Andrew G. Barto and I am stuck on the following question. The value of a state depends on the the values of the ...
1
vote
1answer
4k views

finding themes from text documents

I have a text documents that contain 1000s of abstracts from medical whitepapers. I want to find themes from that text. Any suggestions other than text clustering since clustering helped me to find ...
1
vote
1answer
172 views

Reinforcement Learning - How are these state values in MRP calculated?

This is a question from the book an Introduction to RL, page 125, example 6.2. The example compares the prediction abilities of TD(0) and constant $ \alpha $ MC when applied to the below Markov ...
2
votes
1answer
279 views

What are the differences between Reinforcement Learning (RL) and Supervised Learning?

What is the difference between Reinforcement Learning (RL) and Supervised Learning? Does RL hava more difficulty in finding a stable solution? Does Q-learning have more difficulty in finding a ...
2
votes
1answer
4k views

What is the relationship between MDP and RL?

What is the relationship between Markov Decision Processes and Reinforcement Learning? Could we say RL and DP are two types of MDP?
16
votes
5answers
13k views

Python library to implement Hidden Markov Models

What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before. Alternatively, is there a more ...
1
vote
1answer
233 views

Should reinforcement learning always assume (PO)MDP?

I recently just started learning reinforcement learning and learned that reinforcement learning algorithms work under the assumption of MDP or POMDP. However as I read A3C and recent vision based deep ...
1
vote
2answers
47 views

Equations in “Intoduction to RL”: What is the meaning and difference between E, and E with subscript?

This question is from An introduction to RL, page 78. In the formula below the page, both $\mathbb{E}$ and $\mathbb{E_\pi}$ are mentioned. Could you help me understand the difference between ...
1
vote
1answer
102 views

MDP - RL, Multiple rewards for the same state possible?

This question is from An introduction to RL Pages 48 and 49. This question may also be related to below question, although I am not sure: Cannot see what the "notation abuse" is, mentioned ...
7
votes
1answer
740 views

Reward dependent on (state, action) versus (state, action, successor state)

I am studying reinforcement learning and I am working methodically through Sutton and Barto's book plus David Silver's lectures. I have noticed a minor difference in how the Markov Decision Processes ...
3
votes
1answer
67 views

What is the optimal value of a Markov Decision process with Single actions at each state?

I am trying to solve some questions about a MRP (i.e. a Markov Decision process with only one possible action at each state). The setup is as follows: There are two states ($a$ and $b$) stepping to $...
4
votes
3answers
4k views

Markov Chains for sequential data

I am new to Markov chains and HMM and I am looking for help in developing a program (in python) that predicts the next state based on 20 previous states (lets say 20 states in last 20 months). I have ...
2
votes
1answer
38 views

Using Policy Iteration on an automaton

I've read many explanation on how do to policy iteration, but I can't find an example, so I'm stuck right now trying to figure out to Policy Iteration. The numbers next to each state show the reward ...
3
votes
1answer
23 views

What could be some Classification techniques to classify a tree of webpages given the category of each webpage

I want to perform a website classification task where I have modeled a website as a tree of webpages. I already have a model which can assign categories to the nodes in the tree (webpages). I need ...
1
vote
0answers
18 views

Create a graphical viz of list of elements residing in a column in desired order [closed]

I have a list of elements in a column. Example: UID.................Flow 1............................qwerty, asdfgh, zxcvbn, poiuyt, lkjhgf, mnbvcx 2............................qpwoei, alskdj, ...
6
votes
2answers
569 views

How do I choose a discount factor in Markov Decision Problems?

I'm referring to the gamma in the Value function:
3
votes
1answer
120 views

Using dhmm_em to form the hmm of mfccs' from song clips

I was working on a project on music genre classification and decided to use an hmm to model my data. After extracting the mfccs' from multiple song clips I get multiple matrices of size 13 * x (...
2
votes
0answers
199 views

How to use HMMs for continuous value prediction

I have some time-series data, which I need to use to predict a continuous value for a given time-stamp. I was initially doing it using a Multivariate Regression Model but I later figured that a time-...
4
votes
1answer
638 views

Viterbi-like algorithm suggesting top-N probable state sequences implementation

Traditional Viterbi algorithm (say, for hidden Markov models) provides the most probable hidden state sequence given a sequence of observations. There probably is an algorithm for decoding top-N ...
1
vote
1answer
242 views

How to estimate the transition probabilities for a Markov Chain when time intervals are non-equally spaced

I've been given a dataset with a number of observable states. I am trying to apply a Finite State Markov Chain to model the system, but I found that I can't estimate the transition probabilities if ...
5
votes
0answers
793 views

Comparing transition matrices for Markov chains

I have a population, each unit of which exists in one of several states that change over time. I am using first-order Markov chains to model these state transitions. My population can be segmented ...
1
vote
0answers
51 views

Suggestions the way to start [closed]

One of our famous mathematicians, James Simons, used an extension of the Baum-Welch algorithm to 'crack' the wall street when he started trading on the stock market. Now, as Google, all informations ...
2
votes
1answer
6k views

Markov switching models

What are some reference sources for understanding Markov switching models?
1
vote
0answers
505 views

HMM - Matlab for data set to detect anomaly

I have a dataset of oil temperatures. The time series consist of 100 hours of measurement at every second. There is an anomaly in the data that I would like to detect using Hidden Markov Models (HMM). ...
2
votes
1answer
101 views

How can I rank paths through an HMM?

I have a profile hidden Markov model that I use to identify all instances of a user-defined pattern of symbols in a long sequence of symbols. I use the Viterbi algorithm to find the most probable path ...
2
votes
1answer
2k views

emission probability using hmmlearn package in python

I am learning hmm and try to implement it in Python hmmlearn package(http://hmmlearn.github.io/hmmlearn/hmm.html#building-hmm-and-generating-samples). However I am not quite understand what the ...
1
vote
0answers
1k views

Markov chain modelling?

I am working on a personal loan dataset. For each loan, we recorded its credit status monthly after it was drawn by the borrower. Let's say there were 6 status coded by A-F. My project is to use ...
0
votes
2answers
2k views

Can HMM be used as a binary classifier?

I have some time-series data, which I need to use to predict a binary label for a given time-stamp. I was initially doing it using a Logistic Regression Model but I later figured that a time-series ...
1
vote
0answers
268 views

Online methods for sequence prediction

A ml beginner here, so please bear with me. If I understand correctly RNNs seem to be the go to method right now for sequence prediction for a given input (single/as a sequence). But I do not have ...