Is there any other way to partition a large file that does not fit into memory so it can be fed as training data other than using spark? or hadoop?
Yes, of cause. But, it's insignificant, because Spark and Hadoop are better.
This is my idea. Suppose that your memory can take in 100,000 examples. So splitting your data set to files with size lower than 100,000.
The key and most complex step is how to train classify with those data. Good luck, For Gradient descent series optimization algorithms (GB, SGD and so on), most algorithms (SVM, GBDT, Bayes, LR, deeplearn and so on) support this. You could load one file to RAM and fed them to classifier until to find the best parameter.
My code is very simple. Before each iteration, re-shuffling the order of simples and re-splitting data set will boost the classifier.
import numpy as np X = np.random.random((100, 2)) y = [1 if x > x else 0 for x in X] from sklearn.linear_model import LogisticRegression lr_cly = LogisticRegression() def stop_train(X_s, y_s, threshold): scores = [gnb.score(X, y) for X, y in zip(X_s, y_s)] return np.mean(scores) > threshold def iter_train(cly, X, y, threshold=0.99, max_iter=10): X_s = [X[:50, :], X[50:, :]] y_s = [y[:50], y[50:]] iter_times = 0 while iter_times <= max_iter: print "--------------" for X, y in zip(X_s, y_s): cly.fit(X, y) print cly.score(X, y) if stop_train(X_s, y_s, threshold): break iter_times += 1 iter_train(lr_cly, X, y)