I am trying to understand the input for the PrefixSpan model (pyspark). I thought in the input RDD, each row should be a list. However, as I looked into the source code, each row is actually a list of list. Why does it need a list of list instead of just list? Thanks!
def trainPrefixSpanModel(
data: JavaRDD[java.util.ArrayList[java.util.ArrayList[Any]]],
minSupport: Double,
maxPatternLength: Int,
localProjDBSize: Int ): PrefixSpanModelWrapper = {...}
You’ve asked your question a while ago, but maybe it helps someone anyway.
In the mllib guide they say to refer to the PrefixSpan paper by Jian Pei and his team, there’s another version fully available here
There, they explain that a sequence is composed of elements which are composed of items: For example “a sequence (a (abc)(ac)d(cf)) has five elements: a , (abc), (ac) , d and (cf) where items a and c appear more than once respectively in different elements.” (taken from the paper)
That's why you need a list of lists.
And note that items within an element shouldn’t follow any particular order (in the paper they are assumed to be sorted alphabetically). So if you are dealing with time series I suppose that the events arriving at the same time (so no particular order) would form the items of an element.
