I'm using a python script to log IO of a grid job.
the log is formatted like this:


Those files are millions of line long. I'm using Spark to generate graphs and detect anomaly in job IO. But before doing that I need to insert the job ID and the jobname to the column making the file :


The thing is I'm new to big Data technologies and I would like to know if using parquet format is it better to put both jobname and jobid
or knowing that I have only 15 different jobname and jobid in the same log is it better to convert it on the fly using SparkSQL and make a join to a very small table with just jobname;jobid and put only the jobid in my log.


1 Answer 1


Your second solution sounds great, and to use it in the most effective way, look for broadcast variables in Spark.

Then, you're using the features available in spark to optimize it.

Reference: https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-broadcast.html

  • $\begingroup$ Thank you, I'll look into this. Working on it for only 2 weeks without a very strong training it might take me a while to manage but it's an interesting feature. $\endgroup$
    – Kiwy
    May 22, 2018 at 14:55
  • $\begingroup$ Don't worry, if you have any further doubts, question me and read stuff. $\endgroup$ May 22, 2018 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.