I'm looking for a Python library to process large files without performance problems. For example I want to transfer large files with hash calculating. Python faces memory problems with such processes. Is there any library that helps? (such pandas in data science)

Any help or documentation would be appreciated.


  • $\begingroup$ What is your problem exactly ? have you tried pickle ? $\endgroup$
    – lcrmorin
    Feb 23 '20 at 15:50
  • $\begingroup$ @lcrmorin my problem is performance. $\endgroup$
    – jojo
    Feb 23 '20 at 16:08
  • $\begingroup$ What do you mean by performance ? time ? memory usage ? $\endgroup$
    – lcrmorin
    Feb 23 '20 at 16:09
  • $\begingroup$ memory and time. $\endgroup$
    – jojo
    Feb 23 '20 at 17:29
  • 1
    $\begingroup$ You can read files by batches. For text files it is file.readlines(chunk_size) until you get EOF. It is also possible that your variables are too big and you could just reduce their size by using better data structures. You did not provide any code, so it is very hard to say, what is really happening. Also using numba and cython significantly increases performance. $\endgroup$
    – keiv.fly
    Feb 23 '20 at 21:01

Browse other questions tagged or ask your own question.