How to scale up algorithm development?

Question

In working on exploratory data analysis, and developing algorithms, I find that most of my time is spent in a cycle of visualize, write some code, run on small dataset, repeat. The data I have tends to be computer vision/sensor fusion type stuff, and algorithms are vision-heavy (for example object detection and tracking, etc), and the off the shelf algorithms don't work in this context. I find that this takes a lot of iterations (for example, to dial in the type of algorithm or tune the parameters in the algorithm, or to get a visualization right) and also the run times even on a small dataset are quite long, so all together it takes a while.

How can the algorithm development itself be sped up and made more scalable?

Some specific challenges:

How can the number of iterations be reduced? (Esp. when what kind of algorithm, let alone the specifics of it, does not seem to be easily foreseeable without trying different versions and examining their behavior)

How to run on bigger datasets during development? (Often going from small to large dataset is when a bunch of new behavior and new issues is seen)

How can algorithm parameters be tuned faster?

How to apply machine learning type tools to algorithm development itself? (For example, instead of writing the algorithm by hand, write some simple building blocks and combine them in a way learned from the problem, etc)

Answer 1

First off, if your data has as many variations (in function of time, context, and others) as to make it hard to apply a single strategy to cope with it, you may be interested in doing a prior temporal/contextual/... characterization of the dataset. Characterizing data, i.e., extracting information about how the volume or specifics of the content varies according to some criteria, usually provides with a better understanding (more consise and precise) than simply inferring algorithms on a brute-force fashion.

So, answering each question:

1. characterization is definitely a means of reducing the number of iterations while trying to select proper algorithms for specific data;
2. if you have a discrete set of criterias on which your data varies, it becomes much easier to scale up solutions, as will know what information you'd gain/lose if simpler/specific solutions were applied;
3. after a characterization, you should be also easier to select parameters, since you'd know what kind of specific data you'd be dealing with;
4. finally, you may use data mining/machine learning algorithms to support this characterization. This includes using:
   • clustering algorithms, to reduce the dimensionality of data;
   • classification algorithms, to help deciding on specific properties the data in function of time/context/... may present;
   • association rules, to predict particular knowledge from the dataset, while also improving/fine-graining the data used for later analysis;
   • and other possible strategies and analyses.

And here is a list of some criterias on which to analyse data, which you may find helpful.

Answer 2

Two things you might find useful:

1. meta-learning to speedup the search for the right model and the optimal parameters. Meta learning consists in applying machine learning tools to the problem of finding the right machine learning tool/parameters for the problem at hand. This for instance this paper for a practical example;

2. gpucomputing to speedup the algorithm on larger datasets. For instance, OpenCV can use GPUs, which are very effective at processing images/videos and can bring 10 to 100 speedups with respect to CPUs. As your computer most probably has a gpucomputing-able GPU, you could gain lots of time using it.

Answer 3

Guessing it's likely you've seen this YouTube demo and the related Google Tech Talk, which is related to these papers:

• P-N Learning: Bootstrapping Binary Classifiers by Structural Constraints
• Tracking-Learning-Detection

And this set of code on GitHub for OpenTLD. If you check the "read me" on GitHub here, you'll notice that author's email (Zdenek Kalal) is listed, so it might be worth sending him an email about your questions, or even inviting him to reply to this question too.