How to speed up optimization using Differential Evolution?

Question

My application is high frequency trading. My data are time series of the bid and ask prices of a stock recorded on every tick (change in price). For each data point I also have a certain indicators that predict the future movement of the price. The indicators have different horizons of the predictions, some being optimal at few second intervals and others few minutes. I need to assign these predictors weights and based on whether the linear combination crosses a threshold, the decision will be taken to buy of sell the stock. So far I have tried the Differential Evolution (DE) method to figure out the weights. I use a black box model with the weights vector $w_i$ and threshold as inputs. For each data point I have a vector of indicators $\alpha _i$. $$ total\_alpha = \sum\alpha _i*w_i $$ If $$ total\_alpha > threshold, BUY $$ Else If $$ total\_alpha < -threshold, SELL $$ The output of the model is the sum of difference between each between the price of each consecutive buy and sell. This output is being optimised by the DE algorithm.

I am having trouble with the computational aspects. My data is very large (~$7 \times 10^8$ rows by 20 columns), and thus the execution time for the DE algorithm is unacceptable.

My question: Is there a better and a faster way to solve this problem?

Answer 1

Do you run your analysis algos in batch or live? Which programing language under which environment do you use?

At a first naive look, i would recommend to parallelize your code as each indicator calculation seems to be independent of the other's.

Answer 2

I cannot comment directly on how to accelerate the DE algorithm you are using, because I have not personally used it. I would however, like to suggest breaking up your problem into two different parts.

The first part is to calculate the weights, by trying to predict the price on the next data-point given prices of "n" previous data-points. The second part of the problem is given the weights that you have determined, get the threshold which will allow you to get the best output ( i.e. sum of earnings).

These two problems are somewhat decoupled (in my opinion), i.e. solving each optimally or suboptimally should provide you with an good overall solution.

The first part is just standard least squares, you can find many references to accelerate it or parallelize different versions of it. In the second part, you can just run a grid search, to find upper-threshold (to sell) and lower-threshold(to buy). Each run in the grid search with a different set of parameters should have O(n) complexity. Grid search itself, can be parallelized very easily.