1
$\begingroup$

I'm using a python script to log IO of a grid job.
the log is formatted like this:

timestamp;fullpath;event;size
1526994189.49;/tmp/folder/;IN_ISDIR;6
1526994189.49;/tmp/folder2/File;IN_ACCESS;36

Those files are millions of line long. I'm using Spark to generate graphs and detect anomaly in job IO. But before doing that I need to insert the job ID and the jobname to the column making the file :

timestamp;fullpath;event;size;jobid;jobname
1526994189.49;/tmp/folder/;IN_ISDIR;6;123456;afakejobname
1526994189.49;/tmp/folder2/File;IN_ACCESS;36;123456;afakejobname

The thing is I'm new to big Data technologies and I would like to know if using parquet format is it better to put both jobname and jobid
or knowing that I have only 15 different jobname and jobid in the same log is it better to convert it on the fly using SparkSQL and make a join to a very small table with just jobname;jobid and put only the jobid in my log.

$\endgroup$
1
$\begingroup$

Your second solution sounds great, and to use it in the most effective way, look for broadcast variables in Spark.

Then, you're using the features available in spark to optimize it.

Reference: https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-broadcast.html

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thank you, I'll look into this. Working on it for only 2 weeks without a very strong training it might take me a while to manage but it's an interesting feature. $\endgroup$ – Kiwy May 22 '18 at 14:55
  • $\begingroup$ Don't worry, if you have any further doubts, question me and read stuff. $\endgroup$ – Felipe Bormann May 22 '18 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.