I have a 2M instances dataset with millions of very very sparse dummy variables created using the hashing trick = hash(orig_feature_name + orig_feature_value)=1. Note that the data is sparse both on rows (every instance has only a limited <100 features=1) and on columns (most features are relevant only to very few instances < 1%)

I discovered that in such sparse scenarios the follow-the-regularized-leader FTRL proximal gradient descent is very popular: paper, reference implementation.

But I'm not sure why shouldn't I prefer a batch gradient descent algorithm? FTRL for all its merits is still an online-learning algorithm that sees one instance at a time.

So what are the advantages and disadvantages of using FTRL vs. a well known sparse least squares algorithm such as LSQR (paper, reference implementation)?

My intuition is that if possible to use all the data for each iteration, we should do it, but I'm not sure...


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.