0
$\begingroup$

I am bit more traditional and I am looking for a statistical method that can help me also do inference or predictions.

I have some table that contains some transitions, where each row of the table correspond to a specific user. If you are into python this is it:

smallData=pd.DataFrame(np.array([[1,2,3,4,5],[4,5,6,7,8],[1,2,3,3,3],[1,2,2,3,3],[1,2,3,5,3]]),columns=range(0,5))
smallData
Out[266]: 
   0  1  2  3  4
0  1  2  3  4  5
1  4  5  6  7  8
2  1  2  3  3  3
3  1  2  2  3  3
4  1  2  3  5  3

Now first row means that the following transitions happen 1-2-3-4-5

this kind of transitions can be of course be visualized as a tree or diagram. So far I have not found any method or library in python or R that can calculate the statistics of transitions. For example for the two transitions [1,2] three transition can happen

  • Transition 3 by 50%,
  • Transition 4 by 25% and
  • Transition 5 by 25%.

These were calculated from the table above.

Even harder will be to predict two steps forward.

I have seen the topic of sequence prediction to be discussed a lot in the machine learning community by using LSTM. I still wonder if in the statistics community are methods to handle such problems and if yes if there are in python, R libraries that can be used for such problem.

$\endgroup$
1
$\begingroup$

it sounds like a stochastic process problem. Have you looked into estimating transition matrices for markov chains?

$\endgroup$
  • $\begingroup$ Thanks for the reply. Yes I have tried it. The problem I see with markov chains is that they model the one-hop transition. I think my problem has also dependencies (memory). I also saw in the literature that people use LSTM to create such generators. Still what I miss is a more statistical-typical approach that can learn and make inference based on memory. $\endgroup$ – Alex P May 18 at 18:07
0
$\begingroup$

The traditional approach would be Conditional Random Fields (CRFs). CRFs models can be designed with a fixed-size "memory", i.e. taking into account the N previous states (where N is a constant).

There are some tools available like CRF++ or Wapiti, but I don't know about python or R libraries.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.