0
$\begingroup$

I'm reading up on how to clean/munge/wrangle data sets in order to run machine learning algorithms on them. Lots of info on how to do the actual wrangling, but a practical detail seems to be glossed over: storage.

My question is quite simple: which is the go-to technology to store/retrieve a large data set in order to run algorithms on it in the most convenient/efficient way possible?

I'm guessing the language in which the algorithms are written is not all that relevant here.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

There are a lot of technologies out there to handle. These are the most popular ones I know of:

For moderate data to operate in chunks- Pandas - pd.read_csv('train.csv', chunksize=chunksize)

For larger data - dask, Hadoop with R

Refer this for various other suggestions.

Improve this answer
$\endgroup$
1
$\begingroup$

Currently Apache Hadoop is one of the popular technologies to store data and Apache Spark is a very popular computational engine to compute/munge large datasets.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.