Let's say I'm building an app like Uber and I want to predict the user's most likely destination based on the user's past history, current latitude/longitude, and time/date.

Here is the proposed architecture -

enter image description here

Let's say I have a pre-trained model hosted as a service. The part I'm struggling with is, how do I get the user features from the database in realtime from the RiderID to be used by the prediction service (XGBoost Model)? I'm guessing a lookup in a SQL database will take too long, considering I have 1M+ users and rides.

Thanks in advance!

  • $\begingroup$ So... you are basically asking how to quickly look up a a user and all of the associated rides? $\endgroup$ – S van Balen Feb 20 '18 at 12:46
  • $\begingroup$ Yes, in a way that is efficient and scalable for a large number of users $\endgroup$ – rohan23 Feb 23 '18 at 9:38
  • $\begingroup$ IMO, a user-specific prediction will be useful only for frequent users for whom you have a lot of history to specifically determine where they might want to go, given the current location, time context. You could consider holding trip information (or a summary of it), for only frequent users, in memory. For the rest of the users, you can use a generic model, although user-agnostic destination prediction is likely to not give much returns, as mentioned in the answer below @The Lyrist. $\endgroup$ – raghu Oct 30 '18 at 9:11

I think most likely the return of your model wouldn’t be worth it given the amount of effort to generalize it enough.

  1. You may have millions of users, but each users need will probably too unique for generalization. I.e., everyone’s commute is so different that what you have learned from other users are probably not applicable to other users. (Unless during rush hours where most people are heading to the core business areas. You don’t need a model for that. )
  2. The trained model will probably be marginally better than logging the users usage history. It probably isn’t worth the effort to train and process these data for what you can gain.
  3. Recent locations is probably good enough for most users and super easy to implement. Your model will probably have a hard time to predict the odd unusual trips anyway.

You are probably better off storing the users current location and query the most likely destination. Or simply look at all the popular trip destinations from that location. A data base with proper index should be able to handle that


It sounds like you are looking for a fast and horizontally scalable database. I would advise you to use a column family database instead of a relational database for storing this kind of data. We are using Google BigTable (BT) for this in a similar use-case. On a 3 node BT cluster with SSD disks we have over 300M records that are fetched by key in 6ms @99 percentile with a load of 1000 requests per second. If the load increases you can just simply add nodes while running to your cluster or remove them. An opensource alternative like Cassandra is even faster in our experience. That database key would be RiderID in your case.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.