# Ways to speed up Python code for data science purposes

Although it might sound like a pure techie question, I would like to know which ways you usually try out, for very data science-like processes, when you need to speed up your processes (given that the data retrieval is not a problem and that it also fits in memory etc). Some of those could be the following, but I would like to receive feedback about any other else:

• good practices as always using Numpy when possible on numeric operations and not loops...
• more good practices like using 'apply', 'applymap'... instead of loops when applying functions to elements of lists, dataframes, etc
• Numba applied on native python loops, numpy arrays...
• multiprocessing with multiprocessing library depending on the available logical cores

This is motivated by the fact that, if we mainly use Python with all its advantages, we do not want to switch to other languages like Scala or Julia, unless there is no alternative.

• It is a myth you need apply or applymap when working with dataframes. For almost all of our data processing problems, there are native pandas or numpy methods which are optimized down to the C or Fortran code. Here's an example of apply being 135x slower than numpy, see Timings section in my answer. That said, there are very rare occasions you need apply and that's when you're doing something not trivial. – Erfan Jan 29 at 12:05
• Here's a very informative answer on when to use apply and why you should try to avoid it. – Erfan Jan 29 at 12:08
• thanks for the great links, I totally agree with that style of answering by means of showing reproducible results :) Nevertheless, I did not mean to always use 'apply/applymap' when dealing with dataframes, but use it instead of pure 'for loops' (I did some experiments in my company showing improvements of about 2 orders of magnitude by using 'from_records(itertools.product...)' to build a cartesian product dataframe and then use applymap with custom function instead of 'for looping'); and of course, numpy-like methods even better when possible. – German C M Jan 29 at 14:05
• your second answer is really good, I will try to keep it in mind as I might have been using 'apply-like methods' too many times ;) – German C M Jan 29 at 14:06
• Even for that there are vectorized methods, see this. Also sklearn has a built in method for this: from sklearn.utils.extmath import cartesian – Erfan Jan 29 at 14:25

• use @tf.function decorator on top of TensorFlow functions to speed up computation