6
$\begingroup$

I am trying to visualize the language tags on github repository data. I have names of 16k github repositories, and all the languages associated with each one of them. Below is a chord diagram I came up with.

However, I find that the chord diagram is not a very good representation of the data because

  1. It does not show one-to-many relationships. For example, most of the repositories with JavaScript will have both html and css. This is represented as JavaScript-HTML and JavaScript-css, but not all together.

  2. The size of the arc does not represent the number of repos in the dataset. For example, the number of repos that use JavaScript is about 6k in my dataset, however, the arc is about 17k. This is because of multiple languages in each repo. For example, if a repo has JavaScript, HTML, CSS and Python, the length of the arc would be 3.

Do you have any suggestions for a better visualization of this data, Thanks. This particular one is a d3 chord diagram, but the visualization can be done using any package (Python or JavaScript or R).

Github repository languages chord diagram

Here is the dataset if you are interested, do this query on Google BigQuery

SELECT
    sample_repos.repo_name,
    sample_repos.watch_count,
    languages.LANGUAGE
FROM
  `bigquery-public-data.github_repos.languages` languages
INNER JOIN
  `bigquery-public-data.github_repos.sample_repos` sample_repos
ON
  languages.repo_name = sample_repos.repo_name
WHERE sample_repos.repo_name IN (
  SELECT repo_name[OFFSET(0)]
  FROM `bigquery-public-data.github_repos.commits`)
ORDER BY sample_repos.watch_count DESC
LIMIT 16000
$\endgroup$
1
  • $\begingroup$ Can you share the dataset? $\endgroup$
    – tuomastik
    Sep 17 '18 at 4:48
3
+50
$\begingroup$

Without seeing the data itself, two visualisations come to mind. One option is to run hierarchical clustering on the repositories, and plot the results as a dendrogram with heatmap (or clustermap). The second would be to generate a force-directed graph of language co-occurrence across the repos.

Dendrogram with Heatmap

This plots all language co-occurrences, similar to an adjacency matrix; this clustermap produced with seaborn is an example of a dendrogram with heatmap:

seaborn - discovering structure in heatmap data

It might be possible to gain insight on predominance/hierarchy from the dendrogram by considering which languages are seen more frequently with others, or not at all. The general co-occurrence of languages should be evident from the heatmap.

The advantage of this visualisation is that it should show relationships between languages more clearly than the chord. In particular, some insight might be gained from the 'hierarchy' of languages within each cluster generated. However, a drawback is that the number of repos is not easily represented.

Force-directed Graph

A visualisation that addresses the drawback of the previous method is the force-directed graph, where structure is derived by using "repulsive forces between nodes and attractive forces between adjacent nodes". This might show the numbers of repos involved more effectively, as nodes appearing more frequently with other nodes are weighted with greater 'mass':

copy of Mike Bostock's d3 les miserables graph

The graph above by Mike Bostock is a great example with code and data, but for your use case it may be improved if language counts were represented by the size of node, similar to this visualisation by Christopher Manning. As an afterthought, this bubble chart might also serve if it could be reworked with edges connecting nodes to show where languages were used together in a repo.


The caveat is that without seeing the data I'm not exactly certain what is possible with this dataset. If you can wrangle the data into counts of which languages appear in each repo, and assume that all languages appearing in the same repo are co-occurrences, then these visualisations should be an option.

$\endgroup$
4
  • $\begingroup$ Thanks a lot for the many ideas, I will look into them, and post in the next few weeks. Question, what does " assume that all languages appearing in the same repo are co-occurrences" mean? Thanks $\endgroup$
    – user61034
    Sep 17 '18 at 18:38
  • $\begingroup$ Good luck! To clarify that sentence - it will confuse the plots as they're presented above if you were in a situation where a repo had multiple languages tagged and you wanted to distinguish sub-groupings or occurrences within that repo. So I just wanted to note the assumption that if languages are tagged together in a repo, then the clustermap and graphs (as seen above) will treat them all as linked. $\endgroup$
    – redhqs
    Sep 17 '18 at 21:06
  • 1
    $\begingroup$ I have not implemented this completely yet. But thanks for the ideas! $\endgroup$
    – user61034
    Sep 20 '18 at 13:26
  • 1
    $\begingroup$ just FYI, I have the visualizations here: sandhya-sago.github.io/github_repo_analysis $\endgroup$
    – user61034
    Oct 18 '18 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.