I plan to scrape some forums (Reddit, 4chan) for a research project. We will scrape the newest posts, every 10 minutes for around 3 months. I am wondering how best to store the JSON data from each scrape, so that pre-processing (via Python) later would be as simple as possible. My options are the following:
- Dump data from each scrape into a fresh file (timestamp as filename). Resulting in 12,960 files of approx. 150kb each OR
- Maintain 1 single large file, whereby the scraper simply appends the new output to the end of the file. Resulting in 1 file of size approx 1.9Gb after 3 months
Does anyone have any recommendations or warnings from their experience about either approach and how this affected pre-processing? I am cautions that having a pre-processing script work on a larger file might take longer to process but then again, opening and closing thousands of files will also be time-consuming.