1
$\begingroup$

I am trying to create a predictive model using linear regression with a dataset that has 157673 entries.

The data (in a csv file) is in such format:

Timestamp,Signal_1,Signal_2,Signal_3,Signal_4,Signal_5
2021-04-13 11:03:13+02:00,3,3,3,12,12

My current code:

filename = 'test.csv'
df = pd.read_csv(filename , parse_dates=['Timestamp'], header=0)
df['Timestamp'] = pd.to_numeric(pd.to_datetime(df['Timestamp']))
u, v, w, x, y, z = df.values.T

X = np.asarray([v, w, x, y, z])
Y = np.asarray([u, u, u, u, u])
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

lineReg = LinearRegression()
lineReg.fit(X_train, y_train)
print('Score: ', lineReg.score(X_test, y_test))
print('Weights: ', lineReg.coef_)

When printing out the shape of both X and Y it is (5, 157673) (after putting u 4 more times in the Y array, since it would otherwise give the error ValueError: Found input variables with inconsistent numbers of samples: [5, 1]).

However now I am running into the error MemoryError: Unable to allocate 185. GiB for an array with shape (157673, 157673) and data type float64.

Why is that? There must be a mistake somewhere and if not, why is it suddenly in the shape of (157673, 157673) instead of (6, 157673) ?

$\endgroup$
0
$\begingroup$

quick fix would be to change the data format - I can't see how your data looks like so my suggestion stay theoretical without example

  1. float64 is the most expensive one. using float32 or float16 - depending how much precision point you need. If they are just integer depending on the range you can use even int8 (if less than 255) - use dtypes parameters in the pd.read_csv function. or convert them later on using astype.

  2. Timestamp - decides what precision of time is needed - if you are fine with days - don't store hours, minutes and others

  3. alternatively usse DataTable https://github.com/h2oai/datatable - Pandas is not the most efficient library when it comes to work with big data. however, I think the first 2 points will fix your issue. Because (157673, 157673) is not too big but 185. GiB sounds too much for it !

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.