I have a csv file of almost 2 million rows and about 20 columns. I am interested in about 40000 of these rows, and in 5 of the columns 'squat', 'bench', 'deadlift', 'bodyweight', 'total'. I'd like to get tuples of these 5 values, but only of the rows that have 'female' in column 'sex' and 'ipf' in column 'parent federation'. Also sometimes one of the first 5 columns has no value specified, I'd like to ignore those.
I can do this by going row by row, checking the condition and then adding a tuple to a list, but this is way too slow and takes up too much memory. Is there a way to expand on this how to take CSV file input in list of tuples using only a couple of columns and filtering?