2
$\begingroup$

I have a large dataset with characters and 90000 intances and I have the error ValueError: array is too big when I have the following code before the plot_kmeans_digits.py code:

data2=list(csv.DictReader(open('C:\diabeticdata.csv', 'rU'))) vec = DictVectorizer() data = vec.fit_transform(data2).toarray()

Do you know how I can solve this error?

Thanks in advance.

$\endgroup$
1
$\begingroup$

The problem is that you're converting a large sparse dataset into a dense array via toarray(). Don't do that :) Why do you need it in dense array format? Try posting more of your code to see if we can figure out better ways to solve the problem. With any large dataset, you want to stay sparse as long as possible and only work with smaller sets of dense data (if dense is needed at all).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.