# Big array and MemoryError: Unable to allocate memory (in Python)

I am trying to create a predictive model using linear regression with a dataset that has 157673 entries.

The data (in a csv file) is in such format:

Timestamp,Signal_1,Signal_2,Signal_3,Signal_4,Signal_5
2021-04-13 11:03:13+02:00,3,3,3,12,12


My current code:

filename = 'test.csv'
df['Timestamp'] = pd.to_numeric(pd.to_datetime(df['Timestamp']))
u, v, w, x, y, z = df.values.T

X = np.asarray([v, w, x, y, z])
Y = np.asarray([u, u, u, u, u])
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

lineReg = LinearRegression()
lineReg.fit(X_train, y_train)
print('Score: ', lineReg.score(X_test, y_test))
print('Weights: ', lineReg.coef_)


When printing out the shape of both X and Y it is (5, 157673) (after putting u 4 more times in the Y array, since it would otherwise give the error ValueError: Found input variables with inconsistent numbers of samples: [5, 1]).

However now I am running into the error MemoryError: Unable to allocate 185. GiB for an array with shape (157673, 157673) and data type float64.

Why is that? There must be a mistake somewhere and if not, why is it suddenly in the shape of (157673, 157673) instead of (6, 157673) ?

1. float64 is the most expensive one. using float32 or float16 - depending how much precision point you need. If they are just integer depending on the range you can use even int8 (if less than 255) - use dtypes parameters in the pd.read_csv function. or convert them later on using astype.