Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set?

Particularly, I still have over 88k variables (features) left and one cannot do much with them without massive amounts of memory. But does doing models on "blocks" lose the interactions occuring between blocks or are there some technique for "aggregating" these?

  • 1
    $\begingroup$ training linear models and Neural networks (nonlinear models) with SGD which stands for stochastic gradient descent. the trick is so simple, you calculate the gradient of the loss on a random batch and update the weights. $\endgroup$ Commented May 10, 2018 at 15:36

1 Answer 1


If "variables" refers to training examples:

You can use Stochastic Gradient Descent (SGD) where each iteration uses one training example.

Or you could use Mini-Batch Gradient Descent where each iteration uses a partition of the training set. SGD is Mini-Batch Gradient Descent where the partition size is one training example.

If "variables" refers to features:

You should use dimensionality reduction to reduce your number of features. For instance, you can use Principal Component Analysis (PCA) to reduce your feature vector size while maintaining high variance. This would also help your models train significantly faster.

  • $\begingroup$ side note: If "variables" refers to features and you have plenty of data points, you may use Auto-encoder to extract more compact representations. see VAE, DAE. SAE, CAE $\endgroup$ Commented May 10, 2018 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.