6
$\begingroup$

(Note: Pulled this question from the list of questions in Area51, but believe the question is self explanatory. That said, believe I get the general intent of the question, and as a result likely able to field any questions on the question that might pop-up.)

Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system?

$\endgroup$
  • 1
    $\begingroup$ About how much tweets a "run" are we talking? $\endgroup$ – Johnny000 May 15 '14 at 7:02
  • $\begingroup$ It's hard to know without a range of tweets, 1000, 100,000 full datahose etc. $\endgroup$ – MCP_infiltrator May 15 '14 at 13:01
  • 1
    $\begingroup$ @Johnny000: 500 million Tweets a day; my understanding is that Twitter limits streams to vendors based on trust/need, but to insure the solution covers the current daily averages, the solution should account what is the max, or you're able to reference as the max via a reliable source other than yourself. $\endgroup$ – blunders May 15 '14 at 13:24
  • 1
    $\begingroup$ Here's more information, appear the "firehose" data feed is pricy, so guessing it would be more relevant to limit input to the volume produce via the Twitter Streaming APIs via the Public Stream; guide to processing the data is here. $\endgroup$ – blunders May 15 '14 at 20:48
  • 1
    $\begingroup$ I'd suggest Apache Kafka as message store and any stream processing solution of your choice like Apache Camel or Twitter Storm $\endgroup$ – Konstantin V. Salikhov May 18 '14 at 11:31
3
$\begingroup$

I'd suggest Apache Kafka as message store and any stream processing solution of your choice like Apache Camel or Twitter Storm

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.