How to cluster based on sensor data? - My first data science job

Question

I'm on my first (real), data, programming job. As everyone can imagine, this can be quite hard and I learn a lot from it, given I am a data science student in university. However, I am completely stuck and I want some help.

To give you some background information, this is what is going on: I am expected to build a model which can cluster people (based on care needed) given data, which is called a 'care-level'. This data are profile settings of users, sensor data (movement, doors, smoke, etc.), and alarm data (panic, smoke, inactivity, etc.). The alarm and sensor data has the timestamp of the event. I have data for 2 months available. In other words, I am expected to build a clustering model of people within those 2 months. Additionally, I don't have any data on their current care-level.

I restructered the data to be grouped on ID, year, month, time-of-day (night, morning, afternoon, evening). This grouped data is linked to every type of sensor, one-hot encoded, and then counted for every ID, year, month, time-of-day respectively. This looks like the following: structure of data, shape: (4688, 10) Disclaimer: for privacy concerns, I have blurred the entries. The prefix 'type_' stands for the type of sensor.

Right now, I have troubles to continue. I have my eyes on the goal, however, I have no clue how to get there anymore. I tried fitting a KMeans model, but I am just not able to interpret the model and if the clusters it 'finds' make any sense. You can basically say it is useless. What do you guys recommend to help me? It would be great if someone can give me some advice!

Cheers!

Answer 1

First of all: Congratulations to your job!

It looks to me as if you have two challenges at hand:

1. Creating a good clustering on time series from multiple sensors
2. Interpreting the clustering and finding a meaningful description of the clusters.

Well, that sounds like an interesting task. Be aware that - as for most data science projects - a good understanding of your data might be crucial (and stackexchange can't do that for you). But there are some ideas that we can give, here:

Clustering of time series

To my understanding, you are clustering people. So all data of one person, collected over a period of time counts as one sample. This means, for one person, there are time series of sensor data.

Typically, one would transform the time series in a fixed-length vector to prepare for the clustering. This could either be done

• by extracting meaningful features (amount of movement, number of alarms, ...), or
• by embeddings. An embidding is trained unsupervised to find a good (but nit necessarily meaningful) transformation of a time series into a fixed-length vector space. Word-Embeddings are prominent examples for text, but there are also embiddings suitable for your data.

In both cases, clustering (independent of the algorithm) depends on a distance measure. Often, the Euclidean distance is used, but there are other options. I suggest that you (perhaps together with your colleagues who know about the data) take some time to consider when two sample are considered similar. If this is not possible, at least a normalization of the features should be done.

If for example the movement is given in meters, it might consist of much greater numbers then the count of the alarms. Without normalization, the movement would dominate the clustering.

These preprocessing steps, that will transform each sample data into a fixed length vector with (more or less) meaningful distances between these vectors, are at least as important as the choice of the clustering algorithm (probably much more important). Now you can apply the clustering algorithms of your choice.

Interpreting the clustering

This is a challenging task. It depends on what kind of interpretation is expected, whom the interpretation addresses (other data scientists, a domain expert, ...) and for which purpose it is used.

Note that each of your training samples will be mapped to one cluster.

• You could start with different samples from each cluster. Often samples are a good starting point to discuss clusters.
• Another option would be to create statistics of meaningful characteristics (mean, std, min, max of something like the number of alarms, the movement, ...)

I hope this helps you to find a direction and some next steps