Let me preface this by saying that I'm a complete beginner to R and data science in general, so my apologies if this is a rather trivial question. I do have a rough idea of what I would like to achieve though, and how the end result should look like. I just don't know which algorithm is best suited for dealing with this kind of data and how to go about implementing it.

I would like to run a clustering algorithm on a data frame with 13369 distinct units, each having just a single variable that is tracked over 15 years. So my data looks something like this:

id  y1  y2  y3  y4  y5  y6  y7  y8  y9  y10 y11 y12 y13 y14 y15

01  0   6   0   3   0   0   0   0   0   0   0   1   1   0   0
02  1   4   1   6   4   3   7   6   3   8   11  10  9   10  10
03  0   0   2   0   5   0   1   0   3   0   0   0   0   0   0
13369 4  9  0   12  5   1   1   0   6   1   2   7   0   0   3

To establish some context for this task, this is the data for authors and the number of publications they have published in each year, spanning over 15 years.

I would like to end up with a dendrogram that depicts various clusters that authors fall into, (I'm guessing) depending on how prolific they were in a certain time period (beginning vs. later years, for example).

I hope I explained everything clearly. Thank you for taking your time and assisting a student in need.


Welcome to the community Blitva!

There are things to clear at the beginning:

  • You may cluster authors based on their time-series but it does not necessarily answer the question you asked. How prolific they were in a certain time-period might need another approach to the problem. In other words, you may cluster the authors but you don't necessarily know what was the reason behind that unless you explicitly define Similarity based on which clusters are formed.
  • Feature Extraction is sometimes needed when you want a specific question to be answered. See my example bellow to understand it more.
  • What does being prolific in a certain time period mean? Imagine a person who published 1 2 1 papers in 3 consecutive years and another one who published 10 20 10 papers in those days. Are they similar? If being prolific is calculated between them they are certainly not similar. But if it is calculated according to each person, then they have completely similar patterns of being prolific! Which one are you looking for?

Now back to the question. My answer is as follows:

If you are looking for similar patterns i.e. rise and fall in the number of publications, go for DTW for calculating the similarities. I am sure there are R implementations of DTW but I am a Python guy ;) After calculating pairwise DTW similarities, you may apply your Hierarchical Clustering to it (or maybe some other algorithms like Spectral Clustering).

Good Luck!

  • $\begingroup$ First of all, thank you for your quick reply, and my apologies for getting back to you only now. I absolutely see what you mean regarding the ambiguity of ''being prolific''. That was poor wording on my part, so let me elaborate. What I am looking for, is to see who are the authors that published in the beginning of their careers, who in the middle, who in the end, and who are the outliers. I have aditional data about the authors (e.g. professional title, field of work, etc.) which I will then compare my findings to, and draw some conclusions based on that. I hope that clarifies my problem. $\endgroup$ – Blitva Apr 1 '19 at 15:55
  • $\begingroup$ Sure! Thanks for the explanations. Then as I guessed, you are looking for similar patterns. As you have 3 periods in your mind, you may take it into account. I will update my answer soon! $\endgroup$ – Kasra Manshaei Apr 2 '19 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.