3
$\begingroup$

I have a dataset from some health institues. The data contains information about malaria cases on a 52 week range. The dataset has 52 columns, one for each week and about 16 rows one for each hospital reporting the number of cases in the particular week diagnosed in that hospital. Example of the dataset with 9 weeks entries :-

SAD Lakhwar 0   0   0   0   0   1   4   3   1
Rural Health Center 2   0   0   6   0   2   0   2   2
Herbertpur Christian Hospital   1   0   1   0   2   0   1   0   1

I have used Hierarchical and K-Means clustering for both determining clusters of hospitals as well as clusters of weeks but my real goal is to cluster in such a manner that outbreaks can be detected using data from consecutive weeks and at the same time the cluster of hospitals in which the outbreak is noticed is also found out.

The techniques I have used till now find me clusters where in some cases weeks are away from each other, for example the 7th and 37th week fall into the same cluster as can be seen in the output given below but I want to achieve continuity in weeks because outbreaks span over a few weeks I understand the reason of the results I am getting but I want continuity, if anyone could help.

The result of trying to cluster the weeks into 4 clusters using k means

Week No x
1   2
2   4
3   4
4   2
5   4
6   2
7   1
8   2
9   2
10  2
11  2
12  4
13  1
14  4
15  4
16  4
17  1
18  4
19  1
20  4
21  1
22  1
23  1
24  1
25  1
26  1
27  1
28  1
29  1
30  1
31  1
32  3
33  1
34  1
35  1
36  1
37  1
38  3
39  3
40  3
41  3
42  3
43  3
44  3
45  3
46  3
47  3
48  3
49  3
50  1
51  4
52  4

Dput of the data

structure(list(V1 = structure(c(13L, 15L, 6L, 10L, 3L, 17L, 12L, 
1L, 2L, 11L, 4L, 14L, 8L, 9L, 7L, 5L), .Label = c("CHC Sahaspur", 
"Comb. Hosp. Premnagar", "Doon Hospital", "FRI Hospital", "Herbertpur                     Christian Hospital", 
"HIHT Jollygrant", "Kalindi Hospital", "MAX Hospital", "PHC Herbatpur ", 
"PHC Kalsi", "PHC Rajawala", "Rural Health Center", "SAD Lakhwar", 
"Shubharti Hospital", "SPS Rishikesh", "Total", "Vaish Nursing Home"
), class = "factor"), V2 = c(0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 
0, 0, 0, 0, 0), V3 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
0, 0, 0), V4 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
0), V5 = c(0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0), V6 = c(0, 
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), V7 = c(1, 0, 0, 
0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0), V8 = c(4, 0, 0, 0, 0, 
0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0), V9 = c(3, 0, 0, 0, 0, 0, 2, 
0, 0, 0, 0, 0, 1, 0, 0, 0), V10 = c(1, 0, 0, 0, 0, 1, 2, 0, 0, 
0, 0, 0, 0, 0, 0, 0), V11 = c(0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 
0, 0, 0, 0, 0), V12 = c(0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 
0, 0, 0), V13 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
0), V14 = c(0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), 
V15 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V16 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V17 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V18 = c(0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1), 
V19 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V20 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), 
V21 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V22 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2), 
V23 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0), 
V24 = c(2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), 
V25 = c(0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0), 
V26 = c(0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1), 
V27 = c(1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V28 = c(0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0), 
V29 = c(0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0), 
V30 = c(1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), 
V31 = c(0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V32 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), 
V33 = c(0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), 
V34 = c(5, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0), 
V35 = c(1, 0, 22, 0, 1, 1, 0, 0, 0, 0, 0, 38, 0, 2, 0, 0), 
V36 = c(0, 2, 4, 2, 1, 0, 0, 0, 0, 0, 0, 23, 0, 2, 0, 1), 
V37 = c(0, 0, 10, 0, 2, 0, 0, 0, 0, 0, 0, 10, 0, 2, 0, 7), 
V38 = c(1, 2, 2, 1, 2, 0, 0, 0, 0, 0, 0, 16, 2, 2, 0, 7), 
V39 = c(0, 1, 9, 0, 28, 2, 0, 0, 0, 0, 8, 12, 0, 1, 0, 2), 
V40 = c(1, 0, 2, 0, 25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V41 = c(0, 0, 3, 0, 10, 0, 0, 0, 1, 0, 0, 18, 0, 0, 0, 1), 
V42 = c(0, 0, 1, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0), 
V43 = c(0, 1, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V44 = c(1, 0, 1, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V45 = c(0, 0, 9, 0, 6, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 1), 
V46 = c(0, 0, 4, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V47 = c(0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V48 = c(0, 0, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0), 
V49 = c(0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V50 = c(0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V51 = c(0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V52 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), 
V53 = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)), .Names =                 c("V1", 
"V2", "V3", "V4", "V5", "V6", "V7", "V8", "V9", "V10", "V11", 
"V12", "V13", "V14", "V15", "V16", "V17", "V18", "V19", "V20", 
"V21", "V22", "V23", "V24", "V25", "V26", "V27", "V28", "V29", 
"V30", "V31", "V32", "V33", "V34", "V35", "V36", "V37", "V38", 
"V39", "V40", "V41", "V42", "V43", "V44", "V45", "V46", "V47", 
"V48", "V49", "V50", "V51", "V52", "V53"), row.names = c(NA, 
16L), class = "data.frame")
Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

Clustering allows you to find clusters of similar hospitals. In your case, hospitals in the same cluster will have

  • around the same number of patients
  • around the same time

But depending on the number of clusters you specify and your data, you might not get results that feel satisfactory. Moreover, clustering gives you an answer, but it might not be an answer to the question you want answered.

Exemple 1: Hospital 1 at week 1 might be grouped with Hospital 2 at week 1, while having respectively 0 and 5 patients. This might make sense in the model, but might not in real life, where 0 or 5 patients could be very different.

To adress this problem, you might want to standardize your patient count. It will help the model to make sense of how different 0 and 5 patients are.

Exemple 2: Hospital 1 at week 1 might be grouped with hospital 3 at week 12, if they are the only one having 23 patients while every Hospital never gets more than 10.

This is the other problem you are facing with clustering; it helps you find similarities, but similarities might not be helpful. K-means can not fully use the fact that the data at week 2 depends from the data at week 1. It only tries to find how close two points are in the 2-dimensional space formed by the week number and patient count that week.

So you want

  • To detect outbreaks
  • To find the hospital in which the outbreak is happening

Anony-Mousse proposed a very simple way of finding "outbreaks"; Compute the difference in patients from week to week. Here would be a way to elaborate on this:

  • Set week 1 at 0 and week $i$ at (number of patients at week $i$ $-$ the number of patients at week $i-1$)
  • Compute the sum of the change over all hospitals with past week, and use a threshold to determine outbreaks. If there are 10 new patients or more that week, there is an "outbreak"
  • Use the initial number of patient per hospital at the week the outbreak is detected to determine which hospital are touched. Use 2-means for (High/Low), 3-means for (High/Medium/Low), and so on.

This is just a simple example, but should give you an hint on to where to search next.

Additional

  • What is an outbreak? If you can define more clearly what you are searching for, you will have better results in the end. Clustering tries to find structure in the data, and outbreaks might be part of that structure, but you should focus on methods specifically for it.
  • If you want to apply data science/machine learning techniques to a problem, try to formulate the problem in that language. It is a framework to reason about data, but a good portion of the job is to translate your problem in ways it can be handled.
  • Learn about methods specifically designed to handle this type of problems. A good place to start would be Change detection, as pointed out by Anony-Mousse.
Improve this answer
$\endgroup$
0
$\begingroup$

Do you really want clustering here?

Instead, I'd look at segmentation and change detection.

For example, you could compute the absolute (or relative) change from one row to another, and segment at the largest change points.

Improve this answer
$\endgroup$
1
  • $\begingroup$ if you could elaborate your answer a little please $\endgroup$
    – amankedia
    Commented Jan 17, 2016 at 12:19
0
$\begingroup$

If you are trying to find both the clusters of weeks and the clusters of hospitals where outbreaks happen then you might be more successful using

1) a simple moving average filter on the weeks, playing with the number of weeks you are averaging - trying 3-5 weeks as the averaging window.
2) now look for hospitals that have high moving averages in the same weeks

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.