2
$\begingroup$

I have a set of data which indicates purchase transaction of users (~1 million records). User can have more than 1 purchase across time. Data is spread over 6-7 months. Attributes that I have are

  1. userid
  2. merchant
  3. amount
  4. day of purchase
  5. date of purchase
  6. time of purchase

Goal -

Cluster users based on their purchase behaviour, so that I can target set of users (Promotions) based on their cluster behaviour. Eg - I want to send promotions to user who execute XX behaviour

What I have tried -

I tried KMeans Clustering in Python Sklearn for 10,000 sample records. By using "elbow method" I have identified that 6-7 cluster are optimal. Still, Clusters are not really helpful in identifying similar group of users. Clusters does not make sense.

Questions -

1) Am I doing this correctly?

2) What would be the ideal algorithm for this type of use case. Should I be looking in Clustering or something like Recommendation algorithm (Group similar users)

3) Questions/Goal/Target group of users might change over time. How can we handle that?

$\endgroup$
  • $\begingroup$ k-means (and many other clustering algorithms) require the data to be continuous vectors. None of your attributes is really appropriate for this. log(amount) may be suitable, but certainly the merchant id is not a continuous variable... $\endgroup$ – Anony-Mousse Jul 23 '16 at 7:35
  • $\begingroup$ The bad news is that clustering such data does never work. All you'll get out is statistical nonsense. You sure can get results by massaging your data into numbers, but probably a fortune teller will give you better clusters. $\endgroup$ – Anony-Mousse Jul 23 '16 at 7:36
5
$\begingroup$

Big Picture:

First of all, the feature set in your data is pretty sparse and uninteresting, so you should not expect to gain much traction from this problem. Use your human mind to think about the types of clusters that could possibly exist and this may help inform the feature engineering that you should perform. Some people may tend to buy things on payday (15th and 30th of the month), some people might like to shop on Sundays, some people might like sports stores, people who shop on Wednesdays could like to buy woman's clothes, midnight shoppers could tend to focus on "as seen on TV" products. There might be signal there, but with only 6 features (really 5), it might be tough to find it.

Normalization/Standardization

Make sure you standardize your data before using k-means clustering. Otherwise, your distance metric will loose rotational invariance and the clusters will become nonsense from feature to feature.

Feature Engineering:

You will want to do some feature engineering to help your clustering algorithm find signal in the data:

1) Get rid of userid unless it has some meaningful information like it is a proxy for when they first signed up i.e. ask yourself whether this would help the algorithm cluster the data or provide noise to confuse the algorithm. The answer is likely the latter.

2) Do you have the merchant name only? Have you one-hot encoded the merchant list? Can you turn the merchant name into one or more variables with lower cardinality like groupings for food, clothes, household, sporting, etc? Can you find other information on merchants (other data), join it into your data and enhance this feature to make it much more useful?

3) Amount can possibly be teased out a bit, but be conservative with this one. You could possibly flag purchases over $100 or $1000 as is_expensive and is_very_expensive. Also possibly squaring the amount could account for buyer behavior around big ticket items. This might just add noise, so be careful with this one.

4) Day of week should be first turned into a numerical variable 1-7, but then I would also recommend turning cyclic variables into two spatial dimensions. I explain how to do this here, and in addition to time, it can be applied to day-of-week also. I recommend keeping both the 1-7 format and the two dimensional spatial format. Are there other days that you want to specifically highlight? You could, for instance have an is_weekend variable that is 1 for Saturday and Sunday and 0 for other days. You could have another feature for is_boring_day that is 1 for Monday, Tuesday, and Wednesday and 0 for other days. Play with it, think outside the box... this can be a very useful feature but requires some coaxing at times.

5) Date of purchase is pretty loaded in that you can pull out the months and turn them into cyclic variables. You can also pull out the day of the month in a similar way. You can create at payday flag for the 15th and 30th and create a feature like 1/(1+days_since_payday) since people might be more likely to buy on payday. You can add a seasons feature from the date. Finally, you can use raw date in days since your earliest piece of data.

6) Similarly for time, you should turn this into a 2D spatial feature. This will make it easier to pull out the spending behavior of people who shop at certain times of the day.

Clustering Algorithms

k-means can be a bit blunt at times. I suggest you try at least two more clustering algorithms besides k-means. DBSCAN is a very sophisticated clustering tool and works well for noisy data. It also benefits from not having to specify the number of clusters a priori. Experiment with changing the neighborhood size and how much noise is allowed to see how this effects your clustering. Try at least one more clustering type in the scikit arsenal... they are very easy to swap out once you do it once.

Other Methods:

Finally, you can play with trying to other supervised learning methods. Maybe try separating your data into training and testing sets and try predicting purchase amounts based on the other features. Or try classifying the type of purchase based on the other information. Maybe try training a SVM for classification and a Bayesian-ridge regressor for predicting the purchase amount.

Handling changing user data:

Changing data can be handled in a number of different ways. The most obtuse is simply to retrain your model periodically, while either aggregating all of the data, or forgetting some of the older data due to a lack of relevance. The next step in sophistication is to use an online learning algorithm, which continually updates the model based on new data.

In general, just have fun with it.

Hope this helps!

$\endgroup$
  • $\begingroup$ Thanks for the detailed answer. Apprecate it. I will try to add few more points. I did not standardize the data and I will work on doing it. 1) The reason I kept userid is because, it will help me assocaiate back userid with the cluster in which user is classified. Is this incorrect approach? 2) For merchant, I have merchant categories. So probably, will replace merchant id with merchant category $\endgroup$ – Neil Jul 25 '16 at 2:00
  • $\begingroup$ 3) I will create categories of amount. Eg - Cheap, moderate, expensive, very_expensive 4) Day of Purchase in my data, is actually day of week. I am also planning to add is_weekday and is_weekend vatiables. Brilliant explanation of turning time into two spatial dimension. I will follow that. $\endgroup$ – Neil Jul 25 '16 at 2:00
0
$\begingroup$

I think its not a problem of algo, you should do the job of understanding and describing clusters after you got them. What helps me is computing means of variables that are used in clustering and then putting their difference in some other logical ground (like animals or colors or some cartoon heroes), then try to put labels on the clusters that describes them best.

Also, in your case, i should start with making some metrics out of your variables, like turn time of purchase into share of purchases for each customer made in day hours or evening, or converting a day to weekend/working day and so on.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.