2
$\begingroup$

So I'm doing a project on the COVID-19 Tweets dataset from the IEEE port and I plan to analyse the tweets over the time period from March 2020 till date. The thing is there's more than 300 CSVs for each data with each having millions of rows. Now I need to hydrate all of these tweets before I can go and filter through them. Hydrating just 1 CSV alone took more than two hours today.

I wanted to know if there's a more efficient way I could go about this. Is it possible to combine all the CSVs into one and then hydrate that one file for a really long time or do I have to go for a smaller dataset if it's taking this long for each file.

I'm just starting out into dealing with real-life data so you can consider me a true beginner and any help would be appreciated. Thanks!

$\endgroup$
2
  • 1
    $\begingroup$ What exactly does "hydrating a CSV file" mean in your case? $\endgroup$
    – n1k31t4
    Jun 4, 2021 at 21:56
  • 1
    $\begingroup$ Hydrating the Tweet IDs would be more appropriate, that's my bad. Twitter policy doesn't allow anyone to share Tweet texts in the public, so you have to hydrate your Tweet IDs with the Twitter API to get the fresh tweets from the stream. It takes approximately 24 hours to hydrate about 8 million IDs, due to the rate limit set by Twitter. $\endgroup$
    – rick458
    Jun 9, 2021 at 16:23

1 Answer 1

2
$\begingroup$

Warning: I don't have any experience with Twitter API so I don't know if there are any specific limitations.

In general with big data, assuming the goal is to apply a repetitive process, it's preferable to split the data into small chunks so that the process can be distributed on multiple cores, ideally using a computing cluster.

However in my experience there's a trade-off between convenience and speed: splitting the data and re-assembling it (MapReduce) requires more manipulations, which implies either more manual work (and possible errors) or more code to implement. This is why I usually consider a distributed design only if either it's easy to implement or it would be really too long to compute sequentially.

In the case you describe, processing a single CSV containing all the data would take around 300x2=600 hours, which is around 25 days... I would probably try to directly process the files in parallel. For example if you have access to only 8 cores it would take around 300x2/8=75 hours, but if you have access to 40 cores it's only around 15 hours, etc.

An additional advantage of batch processing is that if there is an error during the process, you don't need to re-process the whole data but only the batches which failed.

$\endgroup$
1
  • 1
    $\begingroup$ Hey Erwan, thank you so much for the reply! I understand I could perhaps do the batch processing for regular operations. However, I do need the Twitter API and it does have a limitation of hydrating about 8 Million IDs in 24 hours, so it'd take months if I want to hydrate the whole dataset. I should probably look into specific time periods to shorten down the dataset. $\endgroup$
    – rick458
    Jun 3, 2021 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.