Big Picture:
First of all, the feature set in your data is pretty sparse and uninteresting, so you should not expect to gain much traction from this problem. Use your human mind to think about the types of clusters that could possibly exist and this may help inform the feature engineering that you should perform. Some people may tend to buy things on payday (15th and 30th of the month), some people might like to shop on Sundays, some people might like sports stores, people who shop on Wednesdays could like to buy woman's clothes, midnight shoppers could tend to focus on "as seen on TV" products. There might be signal there, but with only 6 features (really 5), it might be tough to find it.
Normalization/Standardization
Make sure you standardize your data before using k-means clustering. Otherwise, your distance metric will loose rotational invariance and the clusters will become nonsense from feature to feature.
Feature Engineering:
You will want to do some feature engineering to help your clustering algorithm find signal in the data:
1) Get rid of userid unless it has some meaningful information like it is a proxy for when they first signed up i.e. ask yourself whether this would help the algorithm cluster the data or provide noise to confuse the algorithm. The answer is likely the latter.
2) Do you have the merchant name only? Have you one-hot encoded the merchant list? Can you turn the merchant name into one or more variables with lower cardinality like groupings for food, clothes, household, sporting, etc? Can you find other information on merchants (other data), join it into your data and enhance this feature to make it much more useful?
3) Amount can possibly be teased out a bit, but be conservative with this one. You could possibly flag purchases over $100 or $1000 as is_expensive and is_very_expensive. Also possibly squaring the amount could account for buyer behavior around big ticket items. This might just add noise, so be careful with this one.
4) Day of week should be first turned into a numerical variable 1-7, but then I would also recommend turning cyclic variables into two spatial dimensions. I explain how to do this here, and in addition to time, it can be applied to day-of-week also. I recommend keeping both the 1-7 format and the two dimensional spatial format. Are there other days that you want to specifically highlight? You could, for instance have an is_weekend variable that is 1 for Saturday and Sunday and 0 for other days. You could have another feature for is_boring_day that is 1 for Monday, Tuesday, and Wednesday and 0 for other days. Play with it, think outside the box... this can be a very useful feature but requires some coaxing at times.
5) Date of purchase is pretty loaded in that you can pull out the months and turn them into cyclic variables. You can also pull out the day of the month in a similar way. You can create at payday flag for the 15th and 30th and create a feature like 1/(1+days_since_payday) since people might be more likely to buy on payday. You can add a seasons feature from the date. Finally, you can use raw date in days since your earliest piece of data.
6) Similarly for time, you should turn this into a 2D spatial feature. This will make it easier to pull out the spending behavior of people who shop at certain times of the day.
Clustering Algorithms
k-means can be a bit blunt at times. I suggest you try at least two more clustering algorithms besides k-means. DBSCAN is a very sophisticated clustering tool and works well for noisy data. It also benefits from not having to specify the number of clusters a priori. Experiment with changing the neighborhood size and how much noise is allowed to see how this effects your clustering. Try at least one more clustering type in the scikit arsenal... they are very easy to swap out once you do it once.
Other Methods:
Finally, you can play with trying to other supervised learning methods. Maybe try separating your data into training and testing sets and try predicting purchase amounts based on the other features. Or try classifying the type of purchase based on the other information. Maybe try training a SVM for classification and a Bayesian-ridge regressor for predicting the purchase amount.
Handling changing user data:
Changing data can be handled in a number of different ways. The most obtuse is simply to retrain your model periodically, while either aggregating all of the data, or forgetting some of the older data due to a lack of relevance. The next step in sophistication is to use an online learning algorithm, which continually updates the model based on new data.
In general, just have fun with it.
Hope this helps!
log(amount)may be suitable, but certainly the merchant id is not a continuous variable... $\endgroup$