In working on exploratory data analysis, and developing algorithms, I find that most of my time is spent in a cycle of visualize, write some code, run on small dataset, repeat. The data I have tends to be computer vision/sensor fusion type stuff, and algorithms are vision-heavy (for example object detection and tracking, etc), and the off the shelf algorithms don't work in this context. I find that this takes a lot of iterations (for example, to dial in the type of algorithm or tune the parameters in the algorithm, or to get a visualization right) and also the run times even on a small dataset are quite long, so all together it takes a while.
How can the algorithm development itself be sped up and made more scalable?
Some specific challenges:
How can the number of iterations be reduced? (Esp. when what kind of algorithm, let alone the specifics of it, does not seem to be easily foreseeable without trying different versions and examining their behavior)
How to run on bigger datasets during development? (Often going from small to large dataset is when a bunch of new behavior and new issues is seen)
How can algorithm parameters be tuned faster?
How to apply machine learning type tools to algorithm development itself? (For example, instead of writing the algorithm by hand, write some simple building blocks and combine them in a way learned from the problem, etc)