1
$\begingroup$

Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2.0. This input pipeline meets my needs when data is small enough to fit a PC's RAM.

Now I have some large datasets, more than 10GB, some are larger. I also plan to deploy the models in a production environment, which means there will be new data coming everyday. For DNN model training there is distributed strategy of tensorflow 2.0. But for data preprocessing obviously I cannot use pandas, scikitlearn on the large datasets with one PC. It seems to me I need to use a for-loop where I repeatedly fetch a small part of the data and use it for training?

I am wondering what do people typically use in either experiment or production environment for big data preprocessing? Should I use Spark(PySpark) and Tensorflow input pipeline?

$\endgroup$
1
$\begingroup$

Looking at your use case, Dask, H2O, Modin, Koalas and Vaex would better for scaling your data preprocessing apart from Pyspark. They have API's similar to pandas thus porting your existing code would be easier. But you would need to set them for your target environment.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the reply. Do you know if people use these libraries in production environment? $\endgroup$ – Tyler傲来国主 Dec 12 '19 at 7:19
  • $\begingroup$ Yes, People use these in PROD and they are computing friendly. $\endgroup$ – Syenix Dec 12 '19 at 10:10
1
$\begingroup$

If most of your data and machine learning pipeline is in Python, then Dask project for Python is a very good candidate. It allows you to scale (certain types of) data-frame operations to datasets more than memory. The good thing about Dask is that it is also easy to scale down if required. Most of the code stays in Python, you don't pay the serialization overhead (which is paid in PySpark for Python -> JVM).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.