2
$\begingroup$

I have a dataset which contains time-series data of water flow over time. I have a flow meter connected to a kitchen faucet, and I am trying to cluster or classify specific water usage events.

The data is collected every second, and in each row I am given a value for the amount of gallons which are flowing through my flow meter.

For example, I am trying to classify someone washing their hands, filling a teapot, cleaning dishes, etc...

Is this something that I can use a k-NN Classification Approach to cluster these events? If a clustering based approach isn't good, what other method of classified would be good for this type of data?

If I run some experiments, I can classify each event and turn it into a supervised learning problem. But at the moment, none of the water events are classified.

A very abridged version of my dataset looks like the following:

Time Series Water Usage

EDIT

water = pd.DataFrame(shower1)
rng = pd.date_range('2016-09-01 00:00:00', '2016-09-30 23:59:58', freq='S')
water = water.reindex(rng,fill_value=0.0)
water = water['shower1']
df = pd.DataFrame({'time_stamp':rng,'water_amount':water})

starts = (df['water_amount']>0)&(df['water_amount'].shift(1)==0) #find all starts of events
n_events = sum(starts) #total number of events
df.loc[starts,'event_number'] = range(1,n_events+1) #numerate starts from 1 to n
df['event_number'] = df['event_number'].fillna(method='pad').fillna(-1) #forward fill all the values
df.loc[df['water_amount']==0,'event_number']=-1 #set all event numbers to -1 where the water amount is 0

df.groupby('event_number').agg({'time_stamp':'first',
                                    'water_amount':'sum'}) #feature matrix

enter image description here

$\endgroup$
  • 2
    $\begingroup$ For finding nearest neighbors, all samples have to be of the same dimension which doesn't seem to be the case here, so I would discard this method. You need something robust to different time series lengths. Training an HMM could be a solution but it usually works best in a supervised framework (with some labeled data for the training phase)... $\endgroup$ – Eskapp Jan 17 '17 at 19:08
  • $\begingroup$ When you say that all samples have to be of the same dimension, are you saying that since my samples do not represent the same event, that they're of different dimension? Thanks for your response. I'll take a look at HMMs. $\endgroup$ – Gary Jan 17 '17 at 19:56
  • 1
    $\begingroup$ No, I'm saying that k-NN search relies on a distance. And that the distance between a sample of dimension K and a sample of dimension M, with K not equal to M, is not naturally defined. In your example, Event 1 is a 5D sample, Event 2 a 2D sample and event 3 a 3D sample. I do not see how you can come up with a k-NN variant that will allow you to compare these different data samples... The best way is to see them as time-series (as you first said), but them k-NN is not a time-series clustering method (or at least not in its classical form). $\endgroup$ – Eskapp Jan 17 '17 at 20:15
  • $\begingroup$ As your time series don't all have the same length, you have to be mindful when choosing the clustering method so that it is robust to length difference (which is the case of HMMs and of most probably more methods) $\endgroup$ – Eskapp Jan 17 '17 at 20:16
  • $\begingroup$ Do you think the HMM could be used for unsupervised clustering based on the volume or duration of the flow? $\endgroup$ – Gary Jan 17 '17 at 21:34
3
$\begingroup$

It seems pretty clear from looking at the data when an event starts and ends(basically whenever there is a sequence of positive values). So, instead of starting with some complicated models, I'd suggest calculating a few simple features (like length of the event, total amount of water, amount/seconds, time to previous event, time of day in seconds from start of recording) for every event and then try some clustering algorithm on that new data. k-NN might even produce something meaningful. But a statistical summary of the features can probably already give you a better idea of how to further approach this.

EDIT1

import pandas as pd
import numpy as np

rng = pd.date_range('2017-01-01 14:00:00', '2017-01-01 14:01:00', freq='S')
water = [0,0,0.2,0.3,0.4,0,0,0.3,0.2,0.5]*6+[0]
df = pd.DataFrame({'time_stamp':rng,'water_amount':water,'event_number':np.zeros(len(water))})

j = 1
for k in range(len(df)):
    if df.ix[k,'water_amount']== 0:
        df.ix[k,'event_number'] = -1
    else:
        if df.ix[k-1,'water_amount'] > 0:
            df.ix[k,'event_number'] = df.loc[k-1,'event_number']
        else:
            df.ix[k,'event_number'] = j
            j = j+1


df.groupby('event_number').agg({'time_stamp':'first',
                                'water_amount':'sum'}) #feature matrix

EDIT2

rng = pd.date_range('2017-01-01 14:00:00', '2017-01-01 14:01:00', freq='S')
water = [0,0,0.2,0.3,0.4,0,0,0.3,0.2,0.5]*6+[0]
df = pd.DataFrame({'time_stamp':rng,'water_amount':water})

starts = (df['water_amount']>0)&(df['water_amount'].shift(1)==0) #find all starts of events
n_events = sum(starts) #total number of events
df.loc[starts,'event_number'] = range(1,n_events+1) #numerate starts from 1 to n
df['event_number'] = df['event_number'].fillna(method='pad').fillna(-1) #forward fill all the values
df.loc[df['water_amount']==0,'event_number']=-1 #set all event numbers to -1 where the water amount is 0

df.groupby('event_number').agg({'time_stamp':'first',
                                    'water_amount':'sum'}) #feature matrix
$\endgroup$
  • $\begingroup$ Thanks for your response! So you're suggesting that after feature extraction, a k-NN may be more telling, as opposed to trying to apply it to my data as it currently sits. What are some common methods that you would use to accomplish this feature extraction? $\endgroup$ – Gary Jan 18 '17 at 3:02
  • $\begingroup$ what software/language are you working with? you could for example simply loop through the data and extract one event at a time $\endgroup$ – oW_ Jan 18 '17 at 18:36
  • $\begingroup$ I am using Python. $\endgroup$ – Gary Jan 18 '17 at 19:54
  • $\begingroup$ Is it possible to iterate through the column, and have a new column for duration, a new column for volume, etc... and place each event into it's own row with these attributes? Effectively turning this into tabulated data? $\endgroup$ – Gary Jan 18 '17 at 20:05
  • $\begingroup$ you could iterate through the column and tag every entry with an event number. then group by the event number and aggregate by each event. $\endgroup$ – oW_ Jan 18 '17 at 20:25
1
$\begingroup$

Don't forget preprocessing your data.

For example, do feature extractionn

  • total amount of water
  • duration
  • variance
$\endgroup$
  • $\begingroup$ Thanks for your response! Can you elaborate on common methods used to accomplish feature extraction? $\endgroup$ – Gary Jan 18 '17 at 3:01
  • $\begingroup$ This is very data specific. You'll have to do this yourself. $\endgroup$ – Anony-Mousse Jan 19 '17 at 0:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.