# Model to predict based on frequency of occurrence

I have the following dataset

+-----------------------------------+
|  Passenger           |    Trip    |
+-----------------------------------+
| John                 | London     |
| Jack                 | Paris      |
| Joe                  | Sydney     |
| John                 | London     |
| John                 | London     |
| Jill                 | New york   |
| Jim                  | Sydney     |
| Jack                 | Paris      |
| James                | Sydney     |
+-----------------------------------+


And am trying to use scikit library to predict the likelihood of next possible trip of a passenger based on the frequency ( In this case John => London). As a novice am unsure on which model / function to use.

Update 2:

If I have over 10 million records , how different should I approach this problem ?

For something like this, you could go with a simpler approach. One idea is to sample randomly among the cities that a given passenger has visited using the amount of times each city has been visited as probabilites.

Here's a way you could do so. I've added a few more examples to the dataframe so that the application is seen more clearly. Say you instead have:

     Passenger    Trip
0       John     London
1       Jack     Girona
2       Jack      Paris
3        Joe     Sydney
4        Joe  Amsterdam
5        Joe  Barcelona
6        Joe  Barcelona
7       John     London
8       John      Paris
9       Jill    Newyork
10       Jim     Sydney
11      Jack      Paris
12     James     Sydney


You could define a function like the folllowing in order to randomly sample from the existing data in the dataframe:

def random_sample(df, name):
import numpy as np
# group the dataframe by Passenger and count
# the different trips
g = df.groupby('Passenger').Trip.value_counts()
# Make the probabilities add up to 1
freq = g[name] / g[name].sum()
# random destination based on
# its probabilities
random_name = np.random.choice(a=freq.index, size=1,
p = freq.values)[0]
# return likelyhood of next randomly chosen
# destination and destination
return freq[random_name], random_name


Usage

Say we want to select a a randomly samples destination for say Joe and also to know which is the likelihood. Considering that the destinations where Joe has been are:

Trip
Barcelona    2
Amsterdam    1
Sydney       1


We could get for example:

for _ in range(5):
freq, dest = random_sample(df, 'Joe')
print('Chosen destination {} with a probability of {}'.format(dest, freq))

Chosen destination Sydney with a probability of 0.25
Chosen destination Barcelona with a probability of 0.5
Chosen destination Barcelona with a probability of 0.5
Chosen destination Barcelona with a probability of 0.5
Chosen destination Sydney with a probability of 0.25

• Thanks Alex for your answer. For a smaller subset this works perfectly fine. Now my next step is to try this with a large dataset (Say, I have over 10 million records). Will the same approach works in that case ? P.S I have also updated my question with this constraint now. Commented Feb 25, 2019 at 0:48
• For that you could do the groupby only once, so before calling the funcction, and send it as an extra agument. Should be fast enough. Let me know :)
– yatu
Commented Feb 26, 2019 at 8:39

The following code worked for the larger dataset !


series_px = df_px_dest.groupby('Passenger')['Trip'].apply(lambda x: x.value_counts().head(1))

df_px = series_px.to_frame()

df_px.index = df_px.index.set_names(['UID', 'DEST'])

df_px.reset_index(inplace=True)

def getNextPossibleDestByUserID(name,df=df_px):
return df.query('UID==@name')['DEST'].to_string(index=False)



My next target is to expose that as an API (using Flask maybe) , Will probably raise a new question for that !!