So I'm trying to transform a pandas dataframe where I have multiple duplicates of a "user" value (my index being a user ID number), demographic info (which shouldn't change) and various purchase variables which the goal is to sum (in a timely fashion)
Right now I'm using a couple for loops which I now is bad practice when using pandas:
for n in users: #Creating an empty array to use to shuffle the new values into #n being the user ID number uservalue = [n] #Add independent variables prods = df[df["User_ID"]==n]["Product_ID"].tolist() uservalue.append(prods) for v in range (0,len(xvars)): uservalue.append(df[df["User_ID"]==n][xvars[v]].mean()) #Adding total spending variables for v in range (0, len(yvars)): uservalue.append(df[df["User_ID"]==n][yvars[v]].sum()) user = pd.Series(uservalue, userdfcols) userdf = userdf.append(user, ignore_index = True)
So, rather than iterating over the entire 500k entry dataframe, putting everything in a container list and then appending that list into a new row, is there a more efficient way to transform by user ID? My feeling is that I should be appending values to the row individually using pandas methods and skip the list but would that even help?