28
$\begingroup$

As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside with a link to the dataset/API, characteristics of the data available were added. Such information should be, and is not limited to:

  • the name of the social network;
  • what kind of user information it provides (posts, profile, friendship network, ...);
  • whether it allows for crawling its contents via an API (and rate: 10/min, 1k/month, ...);
  • whether it simply provides a snapshot of the whole dataset.

Any suggestions and further characteristics to be added are very welcome.

Improve this question
$\endgroup$

7 Answers 7

Reset to default
22
$\begingroup$

A couple of words about social networks APIs. About a year ago I wrote a review of popular social networks’ APIs for researchers. Unfortunately, it is in Russian. Here is a summary:

Twitter (https://dev.twitter.com/docs/api/1.1)

  • almost all data about tweets/texts and users is available;
  • lack of sociodemographic data;
  • great streaming API: useful for real time text processing;
  • a lot of wrappers for programing languages;
  • getting network structure (connections) is possible, but time-expensive (1 request per 1 minute).

Facebook (https://developers.facebook.com/docs/reference/api/)

  • rate limits: about 1 request per second;
  • well documented, sandbox present;
  • FQL (SQL-like) and «regular Rest» Graph API;
  • friendship data and sociodemographic features present;
  • a lot of data is beyond event horizon: only friends' and friends' of friends data is more or less complete, almost nothing could be investigated about random user;
  • some strange API bugs, and looks like nobody cares about it (e.g., some features available through FQL, but not through Graph API synonym).

Instagram (http://instagram.com/developer/)

  • rate limits: 5000 requests per hour;
  • real-time API (like Streaming API for Twitter, but with photos) - connection to it is a little bit tricky: callbacks are used;
  • lack of sociodemographic data;
  • photos, filters data available;
  • unexpected imperfections (e.g., it’s possible to collect only 150 comments to post/photo).

Foursquare (https://developer.foursquare.com/overview/)

  • rate limits: 5000 requests per hour;
  • kingdom of geosocial data :)
  • quite closed from researches because of privacy issues. To collect checkins data one need to build composite parser working with 4sq, bit.ly, and twitter APIs at once;
  • again: lack of sociodemographic data.

Google+ (https://developers.google.com/+/api/latest/)

  • about 5 requests per second (try to verify);
  • main methods: activities and people;
  • like on Facebook, a lot of personal data for random user is hidden;
  • lack of user connections data.

And out-of-competition: I reviewed social networks for Russian readers, and #1 network here is vk.com. It’s translated to many languages, but popular only in Russia and other CIS countries. API docs link: http://vk.com/dev/. And from my point of view, it’s the best choice for homebrew social media research. At least, in Russia. That’s why:

  • rate limits: 3 requests per second;
  • public text and media data available;
  • sociodemographic data available: for random user availability level is about 60-70%;
  • connections between users are also available: almost all friendships data for random user is available;
  • some special methods: e.g., there is a method to get online/offline status for exact user in realtime, and one could build schedule for his audience.
Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Sheer awesomeness! I was actually expecting something like this dissolved into many answers, and you came carrying the whole :D Thanks for the answer. Nice job! :) $\endgroup$
    – Rubens
    Commented Jun 20, 2014 at 18:08
  • 1
    $\begingroup$ I left behind the scope LinkedIn, YouTube, Secret. Maybe other regional networks (QQ?). And would be glad to get any info about them. $\endgroup$
    – sobach
    Commented Jun 20, 2014 at 18:36
8
$\begingroup$

It's not a social network per se, but Stackexchange publish their entire database dump periodically:

You can extract some social information by analyzing which users ask and answer to each other. One nice thing is that since posts are tagged, you can analyze sub-communities easily.

Improve this answer
$\endgroup$
6
$\begingroup$

A good list of publicly available social network datasets can be found on the Stanford Network Analysis Project website:

SNAP datasets

The site contains internet social network data (Facebook, Twitter, Google Plus), Citation networks for academic journals, co-purchasing networks from Amazon and several others kinds of networks. They have directed, undirected, and bipartite graphs and all datasets are snapshots that can be downloaded in compressed form.

Improve this answer
$\endgroup$
5
$\begingroup$

An example from germany: Xing a site similar to linkedin but limited to german speaking countries.

Link to it's developer central: https://dev.xing.com/overview

Provides access to: User profiles, Conversations between users (limited to the user itself), Job advertisings, Contacts and Contacts of Contacts, news from the network and some geolocation api.

Yes it has an api, but I did not find information about the rate. But it seems to me, that some information is limited to the consent of the user.

Improve this answer
$\endgroup$
4
$\begingroup$

Network Repository has tons of social networks, web graphs, bio and brain networks, etc. Best of all, they also have interactive visual analytic tools to compare/explore the various social networks.

Improve this answer
$\endgroup$
2
$\begingroup$

A small collection of such links can be found at here. Many of them are social graphs.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I thank you very much for posting this reference, but I was expecting the answers here to point a publicly available dataset/API for social network, andalso describe what is provided by such source (either the download rate of posts, or what kind of information about users). As your answer is, I guess it would be very welcome to the list of publicly available datasets we have. $\endgroup$
    – Rubens
    Commented Jun 17, 2014 at 15:12
0
$\begingroup$

Thai text from different social media platforms + sentiment labels (positive, neutral, negative).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.