# Approaches to pre-processing the huge but organised text data, with & without the generators

I've a huge text file, hence I'm reading it line-by-line, applying some basic cleaning, and separately writing the X & Y to 2 different csv files. Further I'm preparing 3 directories for each csv - train, val & test and writing each line as a separate csv to appropriate directories - This aids in using the fit_generator() method conveniently, by reading these files 1-at-a-time and train the model.

The concern is, before training, I've pre-processing steps and performing those on these many files, 1 file at a time, doesn't seem to be a practical approach(it won't be time-efficient as the operations wouldn't be vectorized, besides there would be lot of read/write on disk since storing each processed file is also inevitable), are there any other approaches in dealing with such scenarios? What are the best practices? Are custom generator functions the only way? Appreciate any help.

Update: Also, What if my processed data-set is a coo-matrix? Is there a viable way other than converting it to dense before writing? Moreover, my concern is neither about optimal resource utilization nor about time efficiency, it is more about what are the different ways of handling such scenarios, an example might help.

• How big is "huge" in this context? Both number of lines and typical line size, please? Mar 26, 2020 at 23:28
• Few million records as of now but it would increase a lot very soon, also the line length is around ~8k characters but this won't be a issue. Mar 27, 2020 at 18:11
• Can you define "a lot" as a percentage? Can you give us an upper bound? Mar 27, 2020 at 18:58
• Did you try to use PySpark? Spark is a good solution to deal with huge amount of data. Mar 27, 2020 at 20:56