I have the following dataset

|  Passenger           |    Trip    |
| John                 | London     |
| Jack                 | Paris      | 
| Joe                  | Sydney     |
| John                 | London     |
| John                 | London     |
| Jill                 | New york   |
| Jim                  | Sydney     |
| Jack                 | Paris      |
| James                | Sydney     |

And am trying to use scikit library to predict the likelihood of next possible trip of a passenger based on the frequency ( In this case John => London). As a novice am unsure on which model / function to use.

Update 2:

If I have over 10 million records , how different should I approach this problem ?


2 Answers 2


For something like this, you could go with a simpler approach. One idea is to sample randomly among the cities that a given passenger has visited using the amount of times each city has been visited as probabilites.

Here's a way you could do so. I've added a few more examples to the dataframe so that the application is seen more clearly. Say you instead have:

     Passenger    Trip
0       John     London
1       Jack     Girona
2       Jack      Paris
3        Joe     Sydney
4        Joe  Amsterdam
5        Joe  Barcelona
6        Joe  Barcelona
7       John     London
8       John      Paris
9       Jill    Newyork
10       Jim     Sydney
11      Jack      Paris
12     James     Sydney

You could define a function like the folllowing in order to randomly sample from the existing data in the dataframe:

def random_sample(df, name):
    import numpy as np
    # group the dataframe by Passenger and count 
    # the different trips 
    g = df.groupby('Passenger').Trip.value_counts()
    # Make the probabilities add up to 1
    freq = g[name] / g[name].sum()
    # random destination based on 
    # its probabilities
    random_name = np.random.choice(a=freq.index, size=1, 
                     p = freq.values)[0]
    # return likelyhood of next randomly chosen
    # destination and destination
    return freq[random_name], random_name


Say we want to select a a randomly samples destination for say Joe and also to know which is the likelihood. Considering that the destinations where Joe has been are:

Barcelona    2
Amsterdam    1
Sydney       1

We could get for example:

for _ in range(5):
    freq, dest = random_sample(df, 'Joe')
    print('Chosen destination {} with a probability of {}'.format(dest, freq))

Chosen destination Sydney with a probability of 0.25
Chosen destination Barcelona with a probability of 0.5
Chosen destination Barcelona with a probability of 0.5
Chosen destination Barcelona with a probability of 0.5
Chosen destination Sydney with a probability of 0.25
  • $\begingroup$ Thanks Alex for your answer. For a smaller subset this works perfectly fine. Now my next step is to try this with a large dataset (Say, I have over 10 million records). Will the same approach works in that case ? P.S I have also updated my question with this constraint now. $\endgroup$
    – Maddy
    Commented Feb 25, 2019 at 0:48
  • 1
    $\begingroup$ For that you could do the groupby only once, so before calling the funcction, and send it as an extra agument. Should be fast enough. Let me know :) $\endgroup$
    – yatu
    Commented Feb 26, 2019 at 8:39

The following code worked for the larger dataset !

series_px = df_px_dest.groupby('Passenger')['Trip'].apply(lambda x: x.value_counts().head(1))

df_px = series_px.to_frame() 

df_px.index = df_px.index.set_names(['UID', 'DEST'])


def getNextPossibleDestByUserID(name,df=df_px): 
    return df.query('UID==@name')['DEST'].to_string(index=False)

My next target is to expose that as an API (using Flask maybe) , Will probably raise a new question for that !!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.