Working with HPC clusters

Question

In my university, we have an HPC computing cluster. I use the cluster to train classifiers and so on. So, usually, to send a job to the cluster, (e.g. python scikit-learn script), I need to write a Bash script that contains (among others) a command like qsub script.py.

However, I find this process very very frustrating. Usually what happens is that I write the python script on my laptop and then I login to the server and update the SVN repository, so I get the same python script there. Then I write that Bash script or edit it, so I can run the bash script.

As you see this is really frustrating since, for every little update for the python script, I need to do many steps to have it executed at the computing cluster. Of course the task gets even more complicated when I have to put the data on the server and use the datasets' path on the server.

I'm sure many people here are using computing clusters for their data science tasks. I just want to know how you guys manage sending the jobs to the clusters?

Answer 1

Ask your grid administrator to add your local machine as a "submit host", and install SGE (which we assume you are using, you don't actually say) so then you can qsub from your machine.

OR....

Use emacs, then you can edit on your HPC via emacs's "tramp" ssh-connection facilities, and keep a shell open in another emacs window. You don't say what editor/operating system you like to use. You can even configure emacs to save a file in two places, so you could save to your local machine for running tests and to the HPC file system simultaneously for big jobs.

Answer 2

There are many solutions to ease the burden of copying the file from a local machine to the computing nodes in the clusters. A simple approach is to use an interface that allows multi-access to the machines in the cluster, like clusterssh (cssh). It allows you to type commands to multiple machines at once via a set of terminal screens (each one a ssh connection to a different machine in the cluster).

Since your cluster seem to have qsub set up, your problem may be rather related to replicating the data along the machines (other than simply running a command in each node). So, to address this point, you may either write an scp script, to copy things to and from each node in the cluster (which is surely better addressed with SVN), or you may set up a NFS. This would allow for a simple and transparent access to the data, and also reduce the need for replicating unnecessary data.

For example, you could access a node, copy the data to such place, and simply use the data remotely, via network communication. I'm not acquainted with how to set up a NFS, but you already have access to it (in case your home folder is the same across the machines you access). Then, the scripts and data could be sent to a single place, and later accessed from others. This is akin to the SVN approach, except it's more transparent/straightforward.

Answer 3

Your approach of using a source version repository is a good one and it actually allows you also working on the cluster and then copying everything back.

If you find yourself making minor edits to your Python script on your laptop, then updating your SVN directory on the cluster, why not work directly on the cluster frontend, make all needed minor edits, and then, at the end of the day, commit everything there and update on your laptop?

All you need is to get familiar with the environment there (OS, editor, etc.) or install your own environment (I usually install in my home directory the latest version of Vim, Tmux, etc. with the proper dotfiles so I feel at home there.)

Also, you can version your data, and even your intermediate results if size permits. My repositories often comprise code, data (original and cleaned versions), documentation, and paper sources for publishing (latex)

Finally, you can script your job submission to avoid modifying scripts manually. qsub accepts a script from stdin and also accepts all #$ comments as command-line arguments.

Answer 4

From your question's wording I assume that you have a local machine and a remote machine where you update two files — a Python script and a Bash script. Both files are under SVN control, and both machines have access to the same SVN server.

I am sorry I do not have any advice specific to your grid system, but let me list some general points I have found important for any deployment.

Keep production changes limited to configuration changes. You write that you have to "use the datasets' path on the server"; this sounds to me like you have the paths hardcoded into your Python script. This is not a good idea, precisely because you will need to change those paths in every other machine where you move the script to. If you commit those changes back to SVN, then on your local machine you will have the remote paths, and on and on ... (What if there are not only paths, but also passwords? You should not have production passwords in an SVN server.)

So, keep paths and other setup informations in a .ini file and use ConfigParser to read it, or use a .json file and use the json module. Keep one copy of the file locally and one remotely, both under the same path, both without SVN control, and just keep the path to that configuration file in the Python script (or get it from the command line if you can't keep both configurations under the same path).

Keep configuration as small as possible. Any configuration is a "moving part" of your application, and any system is more robust the less it has moving parts. A good indicator of something that belongs into configuration is exactly that you have to edit it every time you move the code; things that have not needed editing can remain as constants in the code.

Automate your deployment. You can do it via a Bash script on your local machine; note that you can run any command on a remote machine through ssh. For instance:

svn export yourprojectpath /tmp/exportedproject
tar czf /tmp/yourproject.tgz /tmp/exportedproject
scp /tmp/myproject.tgz youruser@remotemachine:~/dev

## Remote commands are in the right hand side, between ''
ssh youruser@remotemachine 'tar xzf ~/dev/yourproject.tgz'
ssh youruser@remotemachine 'qsub ~/dev/yourproject/script.py'

For this to work, you need of course to have a passwordless login, based on public/private keys, set up between your local and the remote machine.

If you need more than this, you can think of using Python's Fabric or the higher-level cuisine.