3
$\begingroup$

I was wondering if anyone had any idea how to solve this problem.

So basically I have a dataset where some person approximately comes at some regular interval and I don't know what that interval is. I need to determine if the person comes in at approximately regular intervals, not necessarily what is the specifically value of the interval. For example if a certain person comes to my house to deliver milk over some period on say

Week 1: Mon, Wed, Fri,

Week 2:Mon, Wed, Fri

Week 3:Mon, Thu, Fri

Week 4:Mon, Wed, Fri

Week 5:Mon, Wed, Fri

Week 6:Mon, Wed, Fri

Week 7:Mon, Wed, Fri

Week 8:Mon, Thu, Fri

So as we can see out of these 8 weeks only in 2 weeks the person didn't come on Wednesday and instead came on Thursday which can be attributed to maybe a holiday the day before. So the solution to this example is that the person does follow a regular pattern.

Similarly this is another example. Say the person came on -

Mon, Thu, Sun, Wed, Sat, Tue, Fri, Mon, Wed, Sun

where this person follows a regular pattern because except for the last Wednesday he comes every fourth day.

This is an example of where a person doesn't follow a pattern, say the person came on

Mon, Wed, Sat, Fri, Thu, Fri, Wed, Fri, Sun, Sun, Sat

I have to do this for hundreds of different people.

Another equivalent problem is if I know which days over a certain period of time (say a month) some person arrives I need to determine if they follow some pattern or not

I thought about trying to fit the data into a sinusoidal curve but I'm not sure if it will work when I have a person who comes every say Mon, Tue, Fri or the 7th of every month or the first and third monday of every month etc.

I'm open to any method as long as it has good accuracy. Also, depending on whichever algorithm you think is best, if possible could you share a link to some code which solves a similar problem so I can get a general idea of how I'm supposed to implement my algorithm. I'm pretty new to machine learning / data science. Thank You!

$\endgroup$

2 Answers 2

1
$\begingroup$

First of all, digitalize your data on a regular time scale with a time delta shorter than the shortest assumed period. For instance, in the case of the milkman, you could create a table with one record per day having the value 1 if the man come and 0 if not. The table would look like a regular time-serie :

day, presence
  1, 1 
  2, 0
  3, 1
  4, 0

Then perform a autocorrelation analysis of your time-serie. This is a mathematical tool often used in signal processing that consists in, as suggested by @MichelGrogan in his comment, calculating the correlation between the original serie and copies of the serie delayed by distinct time deltas (1 day, 2 days, ...). The result can be plotted as values of correlation in function of the lag value and is called an autocorrelation spectrum. Read autocorrelation for more details.

The max values of the spectrum will give you the values of the most important frequencies observed in your data and confirm that pattern exists. If you do not see clear max values, you can conclude there are no pattern in your data.

Here is an example in Python from your data :

import numpy as np
import matplotlib.pyplot as plt

day =['mon','tue','wed','thi','fri','sat','sun', 'mon','tue','wed','thi','fri','sat','sun', 'mon','tue','wed','thi','fri','sat','sun' ]
num =[    1,    2,    3,    4,    5,    6,    7,     8,    9,   10,   11,   12,   13,   14,    15,   16,   17,  18,    19,   20,   21]
pres=[    1.,    0.,   1.,   0.,   1.,   0.,  0.,    1.,   0.,   1.,   0.,    1.,  0.,    0.,   1.,    0.,  0.,    1.,   1.,    0.,   0.]

corr=np.correlate(pres,pres,'full')
lag=[i-(len(corr)/2) for i in range(len(corr))] 
plt.plot(lag,corr/np.max(corr))

And the spectrum plot : spectrum

As you can see there are clear maximum at 2 days and 7 days lags.

$\endgroup$
0
$\begingroup$

It appears as though you are attempting to determine the presence of seasonality in this time series, i.e. a process that repeats itself at regular intervals.

In this regard, it might be a good idea to decompose your time series and check for the presence of seasonality trends.

As an example, this graph in R shows clear seasonality patterns for weather across different seasons:

seasonality

By decomposing the series, this might give visual cues as to whether seasonality exists among visits by individuals at particular times.

$\endgroup$
2
  • $\begingroup$ Thank you. I don't think this is a feasible solution because unless I misunderstood you I won't possibly be able to check each time series for a person visually if I have to to do it for hundreds of people. $\endgroup$
    – maxachmed
    Apr 5, 2019 at 15:43
  • $\begingroup$ Fair point. While I would recommend examining seasonality in any case, one thing you could do is obtain the differences in visits across the dataset, i.e. 2 days, 3 days between each visit, etc. Then, plot these differences using a histogram - if you find that one particular time lag stands out then you could use this as a generalisation. $\endgroup$ Apr 5, 2019 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.