Is there any other way to partition a large file that does not fit into memory so it can be fed as training data other than using spark? or hadoop?

  • $\begingroup$ what is the size of your data? what is the size of memory of your computer? $\endgroup$
    – Hunar
    Mar 27, 2019 at 6:55
  • $\begingroup$ @honar.cs Oh I'm not using actual data. I was just curious $\endgroup$
    – edunlimit
    Mar 27, 2019 at 23:06

1 Answer 1


Yes, of cause. But, it's insignificant, because Spark and Hadoop are better.

This is my idea. Suppose that your memory can take in 100,000 examples. So splitting your data set to files with size lower than 100,000.

The key and most complex step is how to train classify with those data. Good luck, For Gradient descent series optimization algorithms (GB, SGD and so on), most algorithms (SVM, GBDT, Bayes, LR, deeplearn and so on) support this. You could load one file to RAM and fed them to classifier until to find the best parameter.

My code is very simple. Before each iteration, re-shuffling the order of simples and re-splitting data set will boost the classifier.

import numpy as np

X = np.random.random((100, 2))
y = [1 if x[0] > x[1] else 0 for x in X]

from sklearn.linear_model import LogisticRegression

lr_cly = LogisticRegression()

def stop_train(X_s, y_s, threshold):
    scores = [gnb.score(X, y) for X, y in zip(X_s, y_s)]
    return np.mean(scores) > threshold

def iter_train(cly, X, y, threshold=0.99, max_iter=10):
    X_s = [X[:50, :], X[50:, :]]
    y_s = [y[:50], y[50:]]

    iter_times = 0
    while iter_times <= max_iter:
        print "--------------"
        for X, y in zip(X_s, y_s):
            cly.fit(X, y)
            print cly.score(X, y)
        if stop_train(X_s, y_s, threshold):
        iter_times += 1

iter_train(lr_cly, X, y)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.