I have a dataset like this:

> df1
                      date               count
1                  2012-07-01          11.749347
2                  2012-08-01           3.492433
3                  2012-09-01           4.539559
4                  2012-10-01          14.429109
5                  2012-11-01           6.203474
6                  2012-12-01          11.570248
7                  2013-01-01           7.952286
8                  2013-02-01          16.265912
9                  2013-03-01          21.481481
10                 2013-04-01          16.643551
11                 2013-05-01          18.849206
12                 2013-06-01           7.188498
13                 2013-07-01          25.343643
14                 2013-08-01          22.260274
15                 2013-09-01          27.531957
16                 2013-10-01          27.838428
17                 2013-11-01          31.343284
18                 2013-12-01          55.105348

As you can see in the plot, I have increasing and decreasing parts.

enter image description here

My questions now are : 1) How can I detect significant level changes in my data?

I already ran a MinMax function which gives me local mins and maxs, but do you have any other ideas how I could group my data into significant intervals which can be seen by human eye? I am searching for multilpe ways to do so.

  • $\begingroup$ Have a look at the tsoutlier package. $\endgroup$ – user2974951 Oct 11 '18 at 8:48
  • $\begingroup$ Thanks for your comment! I already downloaded the package, but when I try to use tsoutliers(df1) it says, function tsoutliers couldn't be found... $\endgroup$ – hpmurg Oct 11 '18 at 8:55
  • $\begingroup$ The function is locate.outliers(), have a look at the manual cran.r-project.org/web/packages/tsoutliers/tsoutliers.pdf. $\endgroup$ – user2974951 Oct 11 '18 at 8:58
  • $\begingroup$ Do I need to convert my dataframe columns to a specific format? $\endgroup$ – hpmurg Oct 11 '18 at 11:11
  • $\begingroup$ Your data needs to be in ts() format. $\endgroup$ – user2974951 Oct 11 '18 at 11:13

Looking for "outliers" will not give you the appropriate answer because you have a trend. Moreover the trend is not linear, so the answer will depend on the trend model.

What you should do is :

(1) fit the trend. I choose a logistic function with a MAD but an exponential with LSE growth would have lead the same results

(2) compute the residuals $e(t)$ as the observed data minus the model.

(3) search for the outliers on the residuals. I tool Tuckey's fence with $|e(t)|$ greater than 1.5 times the interquartile.

The attached graph shows that obviously you should choose an exponential growth with a multiplicative error, thus minimizing the MAPE and computing the errors in percentage.

enter image description here


I endorse AlainD's answer. I would provide your audience with two charts, a scatterplot, without lines, and a trend plot. I like the loess method. Here's how it looks with ggplot

> library(ggplot2)
> p <- ggplot(data = df1, aes(x=date,y=count)) + geom_point() + geom_smooth()
> p

enter image description here


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.