I want to make the following Python data-processing code more efficient by replacing for loops. Is there any way to vectorize code like this?

  1. I have a DataFrame object df that looks somewhat like:

    names number
    bob 5
    sara 10
    bob 8
    foo 12
    moo 16
  2. I want to subset the DataFrame to find out all the rows associated with each name, and then perform an operation on number. This is what I am doing now:

    for myName in set(df['names']):    
        nameSubset = df.loc[df['names']==myName]    
        operation(nameSubset['number'], **args) 
    '''Basically,perform an operation on the `number` column of nameSubset.'''

Is there any way to make this code run faster? Theoretically, this could be made faster if, instead of running through each myName at a time, the computer could process several myNames at any given moment. I'm not sure how to vectorize/parallelize my code to make this happen though.

  • $\begingroup$ So you want to group_by(names), then do an operation on number - is that an aggregation/summary operation (e.g. sum, max, count) or an operation on each individual number, e.g. compute some value)? $\endgroup$ – smci Dec 12 '16 at 23:08
  • 1
    $\begingroup$ This is all covered by the basic pandas doc, e.g. Group By: split-apply-combine. Please skim the doc. This is not really a data-science question. $\endgroup$ – smci Dec 12 '16 at 23:40

Is that what you want?

In [261]: df
  names  number
0   bob       5
1  sara      10
2   bob       8
3   foo      12
4   moo      16

In [262]: def my_op(ser):
     ...:     return ser.sum()

In [263]: df.groupby('names').agg({'number':my_op})
bob        13
foo        12
moo        16
sara       10
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.