I am trying to create a predictive model using linear regression with a dataset that has 157673 entries.
The data (in a csv file) is in such format:
Timestamp,Signal_1,Signal_2,Signal_3,Signal_4,Signal_5 2021-04-13 11:03:13+02:00,3,3,3,12,12
My current code:
filename = 'test.csv' df = pd.read_csv(filename , parse_dates=['Timestamp'], header=0) df['Timestamp'] = pd.to_numeric(pd.to_datetime(df['Timestamp'])) u, v, w, x, y, z = df.values.T X = np.asarray([v, w, x, y, z]) Y = np.asarray([u, u, u, u, u]) X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True) lineReg = LinearRegression() lineReg.fit(X_train, y_train) print('Score: ', lineReg.score(X_test, y_test)) print('Weights: ', lineReg.coef_)
When printing out the shape of both X and Y it is
(5, 157673) (after putting
u 4 more times in the Y array, since it would otherwise give the error
ValueError: Found input variables with inconsistent numbers of samples: [5, 1]).
However now I am running into the error
MemoryError: Unable to allocate 185. GiB for an array with shape (157673, 157673) and data type float64.
Why is that? There must be a mistake somewhere and if not, why is it suddenly in the shape of
(157673, 157673) instead of
(6, 157673) ?