I have a dataset with 19 columns and about 250k rows. I have worked with bigger datasets, but this time, Pandas decided to play with my nerves.
I tried to split the original dataset into 3 sub-dataframes based on some simple rules. However, it takes a long time to execute the code. About 15-20 seconds just for the filtering.
Any alternative way that will improve the performance of the code?
import pandas as pd #read dataset df = pd.read_csv('myData.csv') #create a dataframe with col1 10 and col2 <= 15 df1 = df[(df.col1 == 10) & (df.col2 <= 15)] df = df[~df.isin(df1)].dropna() #create a dataframe with col3 7 and col4 >= 4 df2 = df[(df.col3 == 7) & (df.col4 >= 4)] df = df[~df.isin(df2)].dropna()
In the end, I have the
df1, df2, df dataframes with the filtered data.