6
$\begingroup$

I have a series of seemingly random data dripping in one value at a time through time. Although it appears to be random, the data forms clusters when certain attributes are analysed which the charts show. I'm trying to avoid the fallacy of seeing patterns that are not there, but across several datasets the data does indeed appear to form clusters through time. I'm interested in trying to:

  • detect the clusters of either the blue squares and / or the white squares in the data I have
  • ignoring data which are outliers
  • predicting the future shape of the clusters as data is dripped in one column at a time. (There appears to be a certain momentum through time of the cluster shapes where the current cluster areas of the new data is in nearby region of the preceding clusters)

They appear to take on regular semi-predictable shapes which (to my eyes at least) follow a momentum from the data before. How could I detect this?

My questions are:

  • is this cluster analysis, and
  • what would be the best way to detect the white squares and / or the blue areas. I'm guessing the blue areas would be the easiest to cluster?
  • the white squares appear fairly close to the previous y-values and may continue at a certain angle up or down in value. How is the best way to predict this?

enter image description here enter image description here enter image description here enter image description here

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ I don't see any clusters in your examples. If you want to answer this question, you will need to formalize it. I.e. what is a cluster, what is not a cluster. What would a random process generate? Then you can estimate or simulate random data to get thresholds. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Mar 3, 2016 at 22:09
  • $\begingroup$ For each of the three tables, what do the x and y dimensions represent? Does its shape change over time? $\endgroup$
    – Emre
    Commented Mar 3, 2016 at 22:18
  • $\begingroup$ @Emre x is time, and y is the value i'm trying to analyse and predict from the earlier samples. $\endgroup$
    – user3791372
    Commented Mar 3, 2016 at 22:25
  • $\begingroup$ What does each bucket in the y dimension contain; a real number? Do empty buckets mean there were no event/observation at that time? Do events "update" the state for the corresponding row (e.g., increment it)? $\endgroup$
    – Emre
    Commented Mar 3, 2016 at 22:27
  • $\begingroup$ @Emre that's correct, the blue squares are an empty bucket, at that moment in time, and the white is the observation that the event took place at that moment for that value and so placed in that bucket. Each column represents a new sample with bucket values being reset to 0 $\endgroup$
    – user3791372
    Commented Mar 3, 2016 at 23:09

2 Answers 2

Reset to default
1
$\begingroup$

I think your process is not really a cluster analysis problem, but instead a time series analysis problem. You have some process that is measuring a value ($Y$ variable) at different points in time ($X$ variable). I would suggest using a Shewhart Control Chart or something similar to understand the process that you have. You should also do analysis on the distribution to ensure it is normal, or use the CLT to enable the data to move towards normality.

Improve this answer
$\endgroup$
1
$\begingroup$

First, I'll summarize my understanding of your query and the graphs provided:

  1. The 4 charts represent time on x-axis and value of a random variable on y-axis.
  2. You want to understand whether the underlying process can be decomposed into erratic components and periodic component, and while doing so be able to predict the value at the next point in time.

This appears to be a time-series problem, as noted by Marcus D. I would suggest, you could try the following approach:

  1. Try decomposing the time-series using Holt-Winters (try both additive and multiplicative models)
  2. Try transforming the variable by taking difference from previous value, log difference, etc. which may yield a variable better suited for such filtering approaches.

The seasonal and trend components should give you insights into any predictability of this process. Sometimes domain-specific hypothesis could give good leads for next steps for the analysis.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.