0
$\begingroup$

I have a preprocessing script (google cloud function) that generates files (stored in Google Drive). I want to load those files in my DW (Big Query).

What are the pros and cons of:

1) Running the preprocessing script, generate the files and then loadthose files

vs

2) Loading data directly from the preprocessing script (skip the file generation, just do a direct insert in the DW from the preprocessing script)

?

I am interested in focusing the question not only in terms of technical stuff and costs, but also in terms of data processing methodology.

I think the question could lead to the dilemma loading online or more in a batch process.

I have added some conclusions of mine as answers. Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks!

$\endgroup$
0
$\begingroup$

Loading the data directly from the preprocessing script to the DW means not storing the results of your processing on storage.

Not storing those results on storage may imply:

  • Needing to do the processing again if for some reason the data is needed again, for a new DW or for a new research.
  • Not saving the data as a data sink and so not contributing to a data lake architecture, where data sinks are just saved for future reusing on "unpredicted" situations.

From a technical perspective, some tradeoffs are mentioned on the Google BigQuery docs on Streaming data, from which the following could be highlighted:

  • Consistency management, as for errors or duplicates.
  • Waiting time until data is available for copy and export operations

Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.