1
$\begingroup$

Although it might sound like a pure techie question, I would like to know which ways you usually try out, for very data science-like processes, when you need to speed up your processes (given that the data retrieval is not a problem and that it also fits in memory etc). Some of those could be the following, but I would like to receive feedback about any other else:

  • good practices as always using Numpy when possible on numeric operations and not loops...
  • more good practices like using 'apply', 'applymap'... instead of loops when applying functions to elements of lists, dataframes, etc
  • Numba applied on native python loops, numpy arrays...
  • multiprocessing with multiprocessing library depending on the available logical cores

This is motivated by the fact that, if we mainly use Python with all its advantages, we do not want to switch to other languages like Scala or Julia, unless there is no alternative.

Improve this question
$\endgroup$
6
  • $\begingroup$ It is a myth you need apply or applymap when working with dataframes. For almost all of our data processing problems, there are native pandas or numpy methods which are optimized down to the C or Fortran code. Here's an example of apply being 135x slower than numpy, see Timings section in my answer. That said, there are very rare occasions you need apply and that's when you're doing something not trivial. $\endgroup$
    – Erfan
    Jan 29, 2020 at 12:05
  • $\begingroup$ Here's a very informative answer on when to use apply and why you should try to avoid it. $\endgroup$
    – Erfan
    Jan 29, 2020 at 12:08
  • $\begingroup$ thanks for the great links, I totally agree with that style of answering by means of showing reproducible results :) Nevertheless, I did not mean to always use 'apply/applymap' when dealing with dataframes, but use it instead of pure 'for loops' (I did some experiments in my company showing improvements of about 2 orders of magnitude by using 'from_records(itertools.product...)' to build a cartesian product dataframe and then use applymap with custom function instead of 'for looping'); and of course, numpy-like methods even better when possible. $\endgroup$
    – German C M
    Jan 29, 2020 at 14:05
  • $\begingroup$ your second answer is really good, I will try to keep it in mind as I might have been using 'apply-like methods' too many times ;) $\endgroup$
    – German C M
    Jan 29, 2020 at 14:06
  • $\begingroup$ Even for that there are vectorized methods, see this. Also sklearn has a built in method for this: from sklearn.utils.extmath import cartesian $\endgroup$
    – Erfan
    Jan 29, 2020 at 14:25

1 Answer 1

Reset to default
2
$\begingroup$

Things I care about a lot:

  • list comprehensions instead of loops

  • use apply + lambda functions when forced to iterate operations on pandas dataframes

  • use @tf.function decorator on top of TensorFlow functions to speed up computation

  • use as much SQL as possible when importing data from databases, to avoid doing the same stuff in Python

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks for your clear answer, although I kind of mentioned and usually do the first two points; the last one is not a problem in my use case; the third one is indeed interesting even if not applying it to deep learning model but to numerical operations blocks $\endgroup$
    – German C M
    Jan 29, 2020 at 11:49
  • $\begingroup$ Cool, yeah you can apply it to numerical computations in general, I didn't think of it. One last note: the decorator works only for TF 2, not for the 1.x $\endgroup$
    – Leevo
    Jan 29, 2020 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.