I am trying to create a predictive model using linear regression with a dataset that has 157,673 entries.

The data (in a csv file) is in such format:

2021-04-13 11:03:13+02:00,3,3,3,12,12

My current code:

    filename = 'test.csv'
    df = pd.read_csv(filename , parse_dates=['Timestamp'], header=0)
    df['Timestamp'] = pd.to_numeric(pd.to_datetime(df['Timestamp']))
    u, v, w, x, y, z = df.values.T
    X = np.asarray([v, w, x, y, z])
    Y = np.asarray([u, u, u, u, u])
    X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)
    lineReg = LinearRegression()
    lineReg.fit(X_train, y_train)
    print('Score: ', lineReg.score(X_test, y_test))
    print('Weights: ', lineReg.coef_)

When printing out the shape of both X and Y it is (5, 157673) (after putting u 4 more times in the Y array, since it would otherwise give the error ValueError: Found input variables with inconsistent numbers of samples: [5, 1]).

However, now I am running into the error MemoryError: Unable to allocate 185. GiB for an array with shape (157673, 157673) and data type float64.

Why is that? There must be a mistake somewhere and if not, why is it suddenly in the shape of (157673, 157673) instead of (6, 157673) ?


2 Answers 2


You are using scikit-learn's LinearRegression which optimizes with OLS (ordinary least squares). OLS requires a lot of memory.

You should switch to scikit-learn's SGDRegressor which uses stochastic gradient descent (SGD) to optimize. SGD uses far less memory than OLS.


quick fix would be to change the data format - I can't see how your data looks like so my suggestion stay theoretical without example

  1. float64 is the most expensive one. using float32 or float16 - depending how much precision point you need. If they are just integer depending on the range you can use even int8 (if less than 255) - use dtypes parameters in the pd.read_csv function. or convert them later on using astype.

  2. Timestamp - decides what precision of time is needed - if you are fine with days - don't store hours, minutes and others

  3. alternatively usse DataTable https://github.com/h2oai/datatable - Pandas is not the most efficient library when it comes to work with big data. however, I think the first 2 points will fix your issue. Because (157673, 157673) is not too big but 185. GiB sounds too much for it !


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.