6
$\begingroup$

(Note: Pulled this question from the list of questions in Area51, but believe the question is self explanatory. That said, believe I get the general intent of the question, and as a result likely able to field any questions on the question that might pop-up.)

Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ About how much tweets a "run" are we talking? $\endgroup$
    – Johnny000
    May 15, 2014 at 7:02
  • $\begingroup$ It's hard to know without a range of tweets, 1000, 100,000 full datahose etc. $\endgroup$
    – MCP_infiltrator
    May 15, 2014 at 13:01
  • 1
    $\begingroup$ @Johnny000: 500 million Tweets a day; my understanding is that Twitter limits streams to vendors based on trust/need, but to insure the solution covers the current daily averages, the solution should account what is the max, or you're able to reference as the max via a reliable source other than yourself. $\endgroup$
    – blunders
    May 15, 2014 at 13:24
  • 1
    $\begingroup$ Here's more information, appear the "firehose" data feed is pricy, so guessing it would be more relevant to limit input to the volume produce via the Twitter Streaming APIs via the Public Stream; guide to processing the data is here. $\endgroup$
    – blunders
    May 15, 2014 at 20:48
  • 1
    $\begingroup$ I'd suggest Apache Kafka as message store and any stream processing solution of your choice like Apache Camel or Twitter Storm $\endgroup$
    – Konstantin V. Salikhov
    May 18, 2014 at 11:31

1 Answer 1

Reset to default
3
$\begingroup$

I'd suggest Apache Kafka as message store and any stream processing solution of your choice like Apache Camel or Twitter Storm

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.