1
$\begingroup$

I am currently writing a package to streamline data analysis for a research lab. There are several different analysis software packages that we use, based out of unix, matlab, and (rarely used) python. A typical data set is about 250GB (raw), and requires at least 4 different preprocessing steps before analysis. The finished product typically ends up taking up about 1TB. The goal of my package is to allow the user to pick and choose which existing package to use for each step before running the analysis, and then the program will execute it without further user intervention. Since the goal is to integrate these different packages, written in different languages, I decided to write the program in bash to make it easy to call the actual analysis scripts no matter what language they are written in.

The program is starting to come along, but it is getting very complex because of the various idiosyncratic expectations and conventions of each analysis package. I realize bash may not be the most suitable language for complex tasks, but I like that it's easy to call scripts in different languages from there, and that it's relatively simple. The program also does a lot of file handling, which bash is good at. On the other hand, I hear it's also very slow, and it gets clunky when things get more complicated.

I'm wondering if bash is the best choice for this task. Does anyone have suggestions for other languages, or combinations of languages, that might be better suited to my needs?

I should note that I am a self-taught programmer and this is my first real programming challenge. I am mostly familiar with bash, matlab, R, and a little bit of python, but I'd like to learning new things too (C maybe?). Also, this is all going to run on unix.

Improve this question
$\endgroup$
6
  • $\begingroup$ I would advise against bash, for reasons similar to the ones you specified. I personally a python fan, because of it's simplicity and versatility. Creating processes and working with files are also really simple, as these the only requirements you've provided. $\endgroup$
    – NirIzr
    Commented Aug 21, 2016 at 19:50
  • $\begingroup$ I did experiment with python a little bit. Calling subprocesses from different packages just is not as clean from python as from bash (with importing different modules and such). Also, I wanted to stay close to the software doing the actual analyses so users familiar with any one of the options wouldn't have to relearn anything in a different language (e.g. they could add their own scripts and call them pretty much as they would within matlab/bash/python). You're right though, it might be worth it for speed and handling of the more complex parts. Maybe I'll give it another shot. Thanks! $\endgroup$
    – user23593
    Commented Aug 21, 2016 at 20:16
  • $\begingroup$ take a look at os.system, that's basically what you get with bash. $\endgroup$
    – NirIzr
    Commented Aug 21, 2016 at 21:43
  • $\begingroup$ Perl 5 is also a good choice, designed to be a glue language with bash-like backward compatibility, or resemblances. If you want to control long-running processes started by few selected users via web pages, consider perl/CGI. Old but useful in a data staging environment. Web server configuration might require a lot of learning, though. - It is less popular, but still possible to construct GUI frontends based on widget libraries (such as Tk) with perl. $\endgroup$
    – knb
    Commented Aug 22, 2016 at 8:37
  • $\begingroup$ Thanks for the suggestions everyone! I decided to stick with bash and write python or perl helper scripts when things get too complicated $\endgroup$
    – user23593
    Commented Aug 23, 2016 at 21:23

1 Answer 1

Reset to default
0
$\begingroup$

If you are mostly stitching up together calls into other software, like unix utilities (awk, grep, sed ...), python, and matlab scripts, bash is just fine or possibly even best for the job to construct simple pipelines and workflows.

It's easy in bash to read user input, store it in variables, then launch other software depending on the set variables. It's perfectly fast enough for that, and nothing else gets any easier.

If you, however, were to use bash for preprocessing itself, like looping through files line by line, packing and unpacking tab-separated values into arrays etc., that would be excruciatingly slow and not recommended.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.