I'm new to data science and I'm currently working on a project to classify electricity consumption profiles.
This consists of electricity meter readings taken from sites on a half-hourly interval over a year (17520 readings in total). I am using Python to analyse the data.
I have settled on the current process of z-normalising the data, applying dynamic time warping on the entire dataset (using DTAIDistance) and then using agglomerative clustering as suggested by answers to similar questions on the site. I have produced the dendrogram below using 'complete' linkage using scipy.
I have some questions regarding the process.
- What is the "best"* metric to determine the quality of clustering (i.e. where do I set the cut off on my dendrogram?) and is there any iterative method to determine the "optimal" cut off? (* I am aware that "best" is quite subjective for this type of clustering)
- Following up on 1. - say if I want to produce an elbow plot from my DTW distance matrix. How would I go about doing this in Python?
- How can I find out if I have selected the correct linkage method? I have discounted Ward linkage as it apparently relies on an Euclidean distance matrix and have plotted dendrogram using other linkage methods, but I am unsure as to assess their suitability for my dataset.
- I have read that z-normalising data is required for most applications of DTW - when is z-normalising not required for DTW?
- Are any other clustering methods I should consider?