I have a preprocessing script (google cloud function) that generates files (stored in Google Drive). I want to load those files in my DW (Big Query).
What are the pros and cons of:
1) Running the preprocessing script, generate the files and then loadthose files
2) Loading data directly from the preprocessing script (skip the file generation, just do a direct insert in the DW from the preprocessing script)
I am interested in focusing the question not only in terms of technical stuff and costs, but also in terms of data processing methodology.
I think the question could lead to the dilemma loading online or more in a batch process.
I have added some conclusions of mine as answers. Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.