Calculating mean of data frame inside a series object

Question

I have a data frame with following structure:

df.columns

Index(['first_post_date', 'followers_count', 'friends_count', 'last_post_date','min_retweet', 'retweet_count', 'screen_name', 'tweet_count',  'tweet_with_max_retweet', 'tweets', 'uid'],
      dtype='object')

Inside the tweets series, each cell is another data frame containing all the tweets of an user.

df.tweets[0].columns

Index(['created_at', 'id', 'retweet_count', 'text'], dtype='object')

I want to perform calculation on the tweets of each users, for example, finding average number of retweets of each user, the tweet with maximum retweets etc.

How can I do these?

Edit link to sample data

Answer 1

In the absence of a MultiIndex (the Right Way$^\mathrm{TM}$), the apply method can do what you want; e.g.

df.assign(
    max_retweet=df.tweets.apply(lambda x: x.retweet_count.argmax('retweet_count')),
    avg_retweet=df.tweets.apply(lambda x: x.retweet_count.mean())
)


       avg_retweet  max_retweet  
26662     0.045476          187  
32316     0.821538          427  
25879     0.633681          583  
43411     0.112465          421  
28840     0.472222            8

Also, don't share data with pickle; it's dangerous.