1
$\begingroup$

Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2.0. This input pipeline meets my needs when data is small enough to fit a PC's RAM.

Now I have some large datasets, more than 10GB, some are larger. I also plan to deploy the models in a production environment, which means there will be new data coming everyday. For DNN model training there is distributed strategy of tensorflow 2.0. But for data preprocessing obviously I cannot use pandas, scikitlearn on the large datasets with one PC. It seems to me I need to use a for-loop where I repeatedly fetch a small part of the data and use it for training?

I am wondering what do people typically use in either experiment or production environment for big data preprocessing? Should I use Spark(PySpark) and Tensorflow input pipeline?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Looking at your use case, Dask, H2O, Modin, Koalas and Vaex would better for scaling your data preprocessing apart from Pyspark. They have API's similar to pandas thus porting your existing code would be easier. But you would need to set them for your target environment.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the reply. Do you know if people use these libraries in production environment? $\endgroup$
    – CyberPlayerOne
    Dec 12, 2019 at 7:19
  • $\begingroup$ Yes, People use these in PROD and they are computing friendly. $\endgroup$
    – Syenix
    Dec 12, 2019 at 10:10
1
$\begingroup$

If most of your data and machine learning pipeline is in Python, then Dask project for Python is a very good candidate. It allows you to scale (certain types of) data-frame operations to datasets more than memory. The good thing about Dask is that it is also easy to scale down if required. Most of the code stays in Python, you don't pay the serialization overhead (which is paid in PySpark for Python -> JVM).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.