Markov Process and transition matrix

I would like to find some good courses but also a quick response on how to model transition matrix given the states.

Imagine having 4 states and the following array [1,2,4,1,3,4,2 etc etc]. What calculations are possible with only an array of states? You can make the array as long as you want, I just gave a random exemple.

Python ; Excel; Blog solutions are welcome

Your problem is unusual in two ways:

• Apparently the states are known, not hidden. Afaik it's much more common that the states are hidden, and only observations are known. This is what Hidden Markov Models deal with.
• There's a single sequence. It can be a problem if the sequence is not long enough to show a representative sample of all the transitions. Also it's unknown whether there are any other possible initial states.

Apart from this, you can easily estimate a transition matrix: just count how many times each pair of states appear next to each other. For example in 1,2,4,1,3,4,2,4,2,2,4,1 we have:

• 1 -> 2: 1
• 1 -> 3: 1
• 2 -> 2: 1
• 2 -> 4: 3
• 3 -> 4: 1
• 4 -> 1: 2
• 4 -> 2: 2

Note: the total number of transitions should be equal to the length of the sequence minus 1.

From this we can calculate every transition probability,it's just the conditional probability of arriving in state $$x$$ given starting point $$y$$, i.e.

$$p(x|y)=\frac{\#(x,y)}{\sum_{y'}\#(x,y')}$$

Here $$\#(x,y)$$ means "number of times that y -> x happens". In the example:

• 1 -> 2: 1/2
• 1 -> 3: 1/2
• 2 -> 2: 1/4
• 2 -> 4: 3/4
• 3 -> 4: 1
• 4 -> 1: 1/2
• 4 -> 2: 1/2
• Thanks for the clarification. I understand now that it is not HMM. However which one do you suggest as the better solution between Markov chain and Markov decision process ? Feb 9 at 8:49
• @GeriLeka I don't know Markov decision processes very well, but it's clear that the choice completely depends on the target task, including what the data represents. In general choosing a method only for technical reasons is often a mistake, imho. Feb 9 at 9:39
• As I stated before, I have those type of data and the target is to predict the next state or at least simulate multiple states in some type of monte carlo and take the average to predict the next state. However it is not my field, that's why I'm asking for help ahhaha Feb 9 at 10:02
• @GeriLeka no, I meant for example what are the states? do they represent words in a text, steps in a manufacturing process, status of player in game... Usually there are similar problems which have been studied and solved before, that's why it's useful to identify the task exactly. Feb 9 at 11:09
• Economic states, these are derived from Growth and Inflation. To make it easier I create those states since it is more simple to predict a state instead of a real number Feb 9 at 15:12