Random forests, GBM or even the newer and fancier xgboost are not the best candidates for binary classification (predicting ups and down) of stocks predictions or forex trading or at least not as the main algorithm. The reason is that, for this particular problem, they require a huge amount of trees (and tree depth in case of GBM or xgboost) to obtain reasonable accuracy (Breiman suggested using at least 5000 trees and to "not be stingy" and in fact his main ML paper on RF he used 50,000 trees per run).
However, some quants use random forests as feature selectors while others use it to generate new features. It all depends on the characteristics of the data.
I would suggest you read this question and answers on quant.stackexchange where people discuss what methods are the best and when to use them, among them ISOMAP, Laplacian eigenmaps, ANNs, swarm optimization.
Check out the machine-learning tag on the same site, there you might find information related to your particular dataset.