I have a dataset of transactional data with customer ID and I want to segment the dataset into groups using cluster analysis. I'm interested in following the evolution of each cluster over time, but since customers have very different behaviours (roughly 50% of the time a customer will change cluster the week after), I was wondering what would be a statistically sound approach. Is it a good idea to train a clustering algorithm every week and look backwards at the weekly evolution of each segment?
You can try
- Dynamic mode decomposition.
- Dynamic Time Warping. Found a nice resource on Towards data science blog.
These two have proven better approaches than PCA for time series clustering.
Happy coding 💻
May be what you were looking for is the Rand index ?
This "is a measure of the similarity between two data clusterings", in other words, if the RI is close to 1 (after repeated clustering over a time window) then your segment are stable.
Run Clustering periodically (say every month). Use the elbow method to make a decision on the best number of clusters (be open to this aspect of the system changing over time). Define / Label what each cluster represents - The centroids of each cluster represents the average behavior of the inmates within the cluster.