Ok, this might seem a trivial question for some and it's not even a question, more like a discussion. I read the rules and I believe it's everything fine, so I'm gonna take my chances...

Here's the thing, I was thinking about how people build a dataset when they want to build a 'propensity to buy' model. For instance, I was checking some Kaggle notebooks, but it's hard to see them in the real world.

This dataset, for example, shows the visitors and buyers in a day. However, it doesn't look like something we would use in the real world, since it uses features like "the product was added to the shopping cart". I mean, unless the model is running 24/7, you won't have this information. And maybe you will get it only after is useful. Also, how would you use this model if it's running every second? These are just some of the things I'm confused, but this is not what I want to question right now.

This post, on the other hand, try to describe how to build a propensity model. It says the model will predict based on last purchases. But how would you build the training dataset? By only getting the purchases? Isn't this dataset biased? And if it's not only about getting the buyers? If not, when should you collect the data from other customers? For instance, if you work in Wallmart and want to know who is more prone to buy a TV, should you only consider the data from buyers? If not, the data being collected on November or December will result in different datasets, since the same person have different features depending on the date.

To summarize everything, let's think about a simple retail store. You have data from the last 5 years, data from all people who went there and bought something. You will release a new dishwasher next month. How would you build a model to predict which customers are going to buy this milk? Or even better, how would you build the training dataset for the task? Again, if you think about getting only the buyers, won't you miss potential customers? If not, is there any way of knowing which is the cutting point for the nonbuyers?


1 Answer 1


This is a long post with many questions, but I will try my best to answer.

Let's start with the terminology: when we say "propensity model", usually we mean predicting future events (e.g. purchase) based on past customers' behavior. That is, we are talking about customers who had already engaged with us in the past (assume we keep the history, of course), not just anyone in the wild.

In marketing, we use the terms remarketing or retargeting to refer to effort of engaging audiences who have already interacted with us, and prospecting for effort on potential (new, previously unseen) audience.

Now, what data do we collect to build propensity model? From a Big Data prospective, I'd say we collect everything we can collect, and figure out what is useful later. Usually, there are 3 categories:

  1. Demographic data. Profile e.g. age, gender, address, whether he/she has a cat etc.

  2. Behavior data. Any interaction between you and each customer.

  3. Conversion data. Usually purchase history; generally what you want the customer to do e.g. buy a thing, signup for an event etc.

Let's look at a real-world example. I used to work in an e-commerce department where we run a website selling a few products. We place web tracking (Google Analytics) on the website to collect each user's profile (e.g. where he is, what device he is using) and behavior (e.g. view a product description page, put a thing in cart, make a purchase etc.).

The conversion rate of our site is usually <1%, i.e. less than 1% of website visitors make a purchase. My task was to boost sales via online advertisement.

We ended up building a propensity model to predict the probability of each customer to come back and make purchase in next few days. If the probability is high, we target online ads to them, hooking them to come back and buy. One feature particularly useful was "put a product in cart, but left without purchase yesterday". Quite obvious why.

(Precisely speaking, this approach has some flaws, but do not affect our discussion here.)

For prospecting, there are many approaches. What we did was via look-alike - essentially we tell Google "here are the list of all our visitors, this 0.5% bought something, the rest 99.5% did not. Please target our ads to those who look like the 0.5% and not like the rest" and let Google handle it with its own look-alike model.

Another example: ever use Facebook or Amazon? Every time you like a post or click a product, a model in backend updates your history log, and recompute which upcoming post or product to show to you. All within seconds, running 24/7 for billions of users globally.

To your last question, predicting for sometime new (e.g. a new dishwasher) is intrinsically hard (the cold start problem).

One approach is to compare with similar products in the past, e.g. Netflix does this to evaluate which new series to invest in.

In marketing, we also utilize traditional research methodology e.g. focus group, closed trials etc. to 'test the water' before full product launch.

Hope this help.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.