I have some doubts about how to deal with high volumes of data. I'm currently working in the data analysis/data science field, so I've had the chance to perform calculations, manipulate data, and obtain some conclusions about data using statistics and machine learning techniques; however, the amount of data that I will have to deal soon is considerably larger than the volume that I've processed until now.

I'm not sure if I can consider this a Big Data problem, I've done some research, and for example, according to this post, my problem is closer to a Medium Data situation.

I'm looking for some insights from the community in terms of tools, procedures, techniques, and other elements that I should consider as a learning stage previous to facing the new tasks that I'll mention later. Something as a career path that I have not found clearly, since most of the documentation that I find focuses more on the analysis itself (theoretically) than the difficulties that one should face when trying to perform this analysis

Current situation

I have been dealing with different volumes of data, in general, I work with data captured during a period of time (generally a week), and the amount is close to 5 GB per day. I've faced some difficulties, which I think that mostly associated with hardware limitations, but I've managed to reduce the information based on some criteria to be able to operate it.

The tools that I used are mostly:

  • Python
  • pandas (perform operations over data, joins, filters, etc.)
  • scikit-learn (mostly clustering, binning, etc.)
  • numpy, matplotlib, etc.
  • Elasticsearch (store information, some queries).

Some issues that I've found are: pandas is not completely suitable when dealing with amounts of data comparable to RAM size; pandas documentation even suggests using other tools to deal with high volumes of data; I'm using an 8 GB of RAM laptop and it almost freezes when dealing with a 4 GB CSV file. Elasticsearch Python API faces some issues when recovering a high number of documents, I've mostly used pagination to deal with it. scikit-learn freezes when performing some clustering. scikit-learn has definitely issues when trying to cluster; my jupyter kernels usually die.

Future situation

In the first stage, I'll have to perform computations (sorting, filtering, aggregations) over static data. The volume will be considerably higher than the one that I've managed until now, with about 10GB per day for a larger time period.

In the second stage, I'll have to deal with some data ingestion, with high transactions per second.

I'd appreciate it if you could give me some insight into the tools, techniques, hardware considerations, and concepts that I should consider to face this new situation.

Best regards


2 Answers 2


The fastest data structures in Python are sets. They are low-level structures, and you can manage your data quickly. To avoid using too much RAM, you can build them progressively with a CSV with a loop. Once all your data is on sets, you should see much faster results.

See also: https://copyprogramming.com/howto/what-makes-sets-faster-than-lists

Otherwise, you can use Cython + Numpy arrays, a C-based library for Python: https://towardsdatascience.com/numpy-array-processing-with-cython-1250x-faster-a80f8b3caa52

Keep in mind that multi-threading could improve the speed greatly. https://towardsdatascience.com/multithreading-multiprocessing-python-180d0975ab29

However, if your 10GB data is complex, creating a MySQL database could be the best option to retrieve useful data quickly.

  • $\begingroup$ Thanks for your answer. The purpose of my question was more about how to deal with data in terms of data engineering / data science topics. For example, be able to perform clustering, calculations or statistical analysis when I have a lot of data. $\endgroup$
    – tms
    Commented Nov 15, 2023 at 18:28
  • $\begingroup$ Generally speaking, taking a random sample of 10% of all data is usually sufficient. I used to work with more than 10GB, and there were two options: either I take a sample to get a general overview, or I cut the data into logical pieces to reach specific objectives. $\endgroup$ Commented Nov 15, 2023 at 21:02
  • $\begingroup$ I understand that this tends to be appropriate to perform statistical analysis. However, as I said, I will have to run some computations over data. An example would be if my data are fuel consumption registers per user ID and city (or something). If I want to sort users based on total fuel consumption and I want to split this information per city, I think that I should run this operation over the complete dataset. In such cases, I think taking a sample might not be appropriate. $\endgroup$
    – tms
    Commented Nov 16, 2023 at 12:56
  • $\begingroup$ I'm afraid I don't know the perfect solution. Some Business Intelligence tools, like Qlik Sense, have a fast memory system to manage 10GB quite fast. This kind of tool let you use Python scripts to use other data science algorithms. Otherwise, you would have to increase your RAM, or use a data base. $\endgroup$ Commented Nov 18, 2023 at 8:08
  • $\begingroup$ Thank you for your insights, they seem interesting, particularly regarding speed considerations. Also, thank you for indicating the references. $\endgroup$
    – tms
    Commented Nov 21, 2023 at 18:44

As it was mentioned, you can use your memory several times more efficiently using other tools (numpy, Redis, C++, R / data.table, etc), but after that you will still hit your laptop RAM limit and then you would need to use or implement some solution that stores your data to the disk.

The problems you face were solved long ago. For large data analysis and production code some database engine is usually used.

Both free solutions (like Postgres, Apache Spark, etc) and paid ones (Google Cloud BigQuery, Microsoft SQL Server, Vertica, kdb+, etc) often go with an associated universe of additional tools that you can use for data processing.

If you are familiar with python and pandas, a professional solution is to use Apache Beam to process your data in parallel (also available as part of some commercial solutions).

  • $\begingroup$ Thank you for your answer. I agree with your comments regarding laptop's RAM limit and the need for disk storing. Do you have any comments regarding this transition? In some situations, I've faced this problem not necessarily when storing data but when operating over it, for example, when some cluster algorithms internally compute distance matrices the use of disk heavily increases. I've read regarding Spark, and it seems interesting. I had not heard about Apache Beam, I'll check its documentation. $\endgroup$
    – tms
    Commented Nov 21, 2023 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.