Model to predict based on frequency of occurrence

Question

I have the following dataset

+-----------------------------------+
|  Passenger           |    Trip    |
+-----------------------------------+
| John                 | London     |
| Jack                 | Paris      | 
| Joe                  | Sydney     |
| John                 | London     |
| John                 | London     |
| Jill                 | New york   |
| Jim                  | Sydney     |
| Jack                 | Paris      |
| James                | Sydney     |
+-----------------------------------+

And am trying to use scikit library to predict the likelihood of next possible trip of a passenger based on the frequency ( In this case John => London). As a novice am unsure on which model / function to use.

Update 2:

If I have over 10 million records , how different should I approach this problem ?

Answer 1

For something like this, you could go with a simpler approach. One idea is to sample randomly among the cities that a given passenger has visited using the amount of times each city has been visited as probabilites.

Here's a way you could do so. I've added a few more examples to the dataframe so that the application is seen more clearly. Say you instead have:

     Passenger    Trip
0       John     London
1       Jack     Girona
2       Jack      Paris
3        Joe     Sydney
4        Joe  Amsterdam
5        Joe  Barcelona
6        Joe  Barcelona
7       John     London
8       John      Paris
9       Jill    Newyork
10       Jim     Sydney
11      Jack      Paris
12     James     Sydney

You could define a function like the folllowing in order to randomly sample from the existing data in the dataframe:

def random_sample(df, name):
    import numpy as np
    # group the dataframe by Passenger and count 
    # the different trips 
    g = df.groupby('Passenger').Trip.value_counts()
    # Make the probabilities add up to 1
    freq = g[name] / g[name].sum()
    # random destination based on 
    # its probabilities
    random_name = np.random.choice(a=freq.index, size=1, 
                     p = freq.values)[0]
    # return likelyhood of next randomly chosen
    # destination and destination
    return freq[random_name], random_name

---

Usage

Say we want to select a a randomly samples destination for say Joe and also to know which is the likelihood. Considering that the destinations where Joe has been are:

Trip
Barcelona    2
Amsterdam    1
Sydney       1

We could get for example:

for _ in range(5):
    freq, dest = random_sample(df, 'Joe')
    print('Chosen destination {} with a probability of {}'.format(dest, freq))

Chosen destination Sydney with a probability of 0.25
Chosen destination Barcelona with a probability of 0.5
Chosen destination Barcelona with a probability of 0.5
Chosen destination Barcelona with a probability of 0.5
Chosen destination Sydney with a probability of 0.25

Answer 2

The following code worked for the larger dataset !

series_px = df_px_dest.groupby('Passenger')['Trip'].apply(lambda x: x.value_counts().head(1))

df_px = series_px.to_frame() 

df_px.index = df_px.index.set_names(['UID', 'DEST'])

df_px.reset_index(inplace=True) 

def getNextPossibleDestByUserID(name,df=df_px): 
    return df.query('UID==@name')['DEST'].to_string(index=False)

My next target is to expose that as an API (using Flask maybe) , Will probably raise a new question for that !!