0
$\begingroup$

I am working on a problem to predict the revenue, a film will generate. Some of the features available in the data set are json collection for the crew, cast which worked in the film. I applied onehotencoding to these columns.
As a result, I have a (3000*1835) sized array. This too I got after extracting only director's data from 'Crew' columns and applying PCA with 60% variance retention.
But, when I apply polynomial regression, I get the below mentioned error:

$\lib\site-packages\sklearn\model_selection_validation.py:532: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details: MemoryError: Unable to allocate 30.2 GiB for an array with shape (2400, 1686366) and data type float64

I am using the code as shown below for polynomial regression:

polyFeature = PolynomialFeatures(degree=2)
linearRegression = LinearRegression()
pipeline = Pipeline([('polyFeature',polyFeature),('linearRegression',linearRegression)])
score = cross_val_score(pipeline,XTrain,YTrain,n_jobs=4,cv=5)

I am using a system with 6 cores, 32 GB RAM.

Improve this question
$\endgroup$
1
  • $\begingroup$ 1. No sense in using polynomial features to binary features (one hot encoded) 2. If you use polynomial features I think it is because you have proven that the relationship between your features and your target is not linear, so why not to use a non linear model like a tree base model instead? (In such case no need for polynomial features) $\endgroup$
    – Multivac
    Mar 24, 2021 at 21:11

1 Answer 1

Reset to default
0
$\begingroup$

You are allocating a float64 (2400, 1686366) array (very big!), no surprise you get a memory error. You say that you have 1835 columns, but the error message says 1686366, so which one is the correct number of columns? If it's 1835 then there's an error before feeding your set to the model.

Also, are you sure you need a float64 data type? You can reduce the memory consumption by using float32 instead of float64 (unless you really need to work with that precision)

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ The reason VK is getting ~1.6m features is not due to an error. It's because using polynomial features increases the feature length a lot ( see here mathoverflow.net/questions/225953/…). $\endgroup$
    – bogovicj
    Feb 27, 2020 at 12:50
  • $\begingroup$ Is there any way to avoid this error, is there any other algorithm that I can use in place of polynomial regression, which takes less memory? $\endgroup$
    – V K
    Feb 27, 2020 at 14:24
  • $\begingroup$ Ok. Then try to convert to float32, it should fit into your memory (your array should be around 15 GB) $\endgroup$
    – black_cat
    Feb 27, 2020 at 15:41
  • $\begingroup$ @black_cat there is no option to do that. Any other ideas $\endgroup$
    – V K
    Mar 7, 2020 at 6:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.