# What are the ways to partition a large file that does not fit into memory so it can later be fed as training data?

Is there any other way to partition a large file that does not fit into memory so it can be fed as training data other than using spark? or hadoop?

• what is the size of your data? what is the size of memory of your computer? – Hunar Mar 27 '19 at 6:55
• @honar.cs Oh I'm not using actual data. I was just curious – edunlimit Mar 27 '19 at 23:06

Yes, of cause. But, it's insignificant, because Spark and Hadoop are better.

This is my idea. Suppose that your memory can take in 100,000 examples. So splitting your data set to files with size lower than 100,000.

The key and most complex step is how to train classify with those data. Good luck, For Gradient descent series optimization algorithms (GB, SGD and so on), most algorithms (SVM, GBDT, Bayes, LR, deeplearn and so on) support this. You could load one file to RAM and fed them to classifier until to find the best parameter.

My code is very simple. Before each iteration, re-shuffling the order of simples and re-splitting data set will boost the classifier.

import numpy as np

X = np.random.random((100, 2))
y = [1 if x[0] > x[1] else 0 for x in X]

from sklearn.linear_model import LogisticRegression

lr_cly = LogisticRegression()

def stop_train(X_s, y_s, threshold):
scores = [gnb.score(X, y) for X, y in zip(X_s, y_s)]
return np.mean(scores) > threshold

def iter_train(cly, X, y, threshold=0.99, max_iter=10):
X_s = [X[:50, :], X[50:, :]]
y_s = [y[:50], y[50:]]

iter_times = 0
while iter_times <= max_iter:
print "--------------"
for X, y in zip(X_s, y_s):
cly.fit(X, y)
print cly.score(X, y)
if stop_train(X_s, y_s, threshold):
break
iter_times += 1

iter_train(lr_cly, X, y)