2
$\begingroup$

I want to make the following Python data-processing code more efficient by replacing for loops. Is there any way to vectorize code like this?

  1. I have a DataFrame object df that looks somewhat like:

    names number
    bob 5
    sara 10
    bob 8
    foo 12
    moo 16
    
  2. I want to subset the DataFrame to find out all the rows associated with each name, and then perform an operation on number. This is what I am doing now:

    for myName in set(df['names']):    
        nameSubset = df.loc[df['names']==myName]    
        operation(nameSubset['number'], **args) 
    
    '''Basically,perform an operation on the `number` column of nameSubset.'''
    

Is there any way to make this code run faster? Theoretically, this could be made faster if, instead of running through each myName at a time, the computer could process several myNames at any given moment. I'm not sure how to vectorize/parallelize my code to make this happen though.

$\endgroup$
2
  • $\begingroup$ So you want to group_by(names), then do an operation on number - is that an aggregation/summary operation (e.g. sum, max, count) or an operation on each individual number, e.g. compute some value)? $\endgroup$
    – smci
    Dec 12, 2016 at 23:08
  • 1
    $\begingroup$ This is all covered by the basic pandas doc, e.g. Group By: split-apply-combine. Please skim the doc. This is not really a data-science question. $\endgroup$
    – smci
    Dec 12, 2016 at 23:40

1 Answer 1

1
$\begingroup$

Is that what you want?

In [261]: df
Out[261]:
  names  number
0   bob       5
1  sara      10
2   bob       8
3   foo      12
4   moo      16

In [262]: def my_op(ser):
     ...:     return ser.sum()
     ...:

In [263]: df.groupby('names').agg({'number':my_op})
Out[263]:
       number
names
bob        13
foo        12
moo        16
sara       10
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.