Organizing a csv file of multiple datasets into a list of Pandas dataframes

Question

I have a csv file, containing results from a Computational Fluid Dynamics (CFD) simulation (a sample of my csv file is attached as a google drive link; file size: 1,392KB). In particular, the csv file has information about multiple streamlines (number of streamlines may reach 1000 depending on the case). All the data for all the streamlines are saved back to back in the csv file (so there is no empty row or something to tell us the end of one streamline and start of the next one). The only way I can distinguish streamlines from each other is that when the value in column "IntegrationTime" is zero, it indicates the start of a new streamline, until we hit another zero in the"IntegrationTime" column which is the start of the next streamline.

I need to read this csv file, and organize its data into a list of Pandas datafreames, like:

streamlineList = [df_for_streamline_1, df_for_streamline_2, ...., df_for_streamline_N]

Note (extras question here): This is not crucial but would be nice to have: if you look at the end of my csv file, you see multiple rows where IntegrationTime is zero (100 rows to be exact). Preferably, I don't need these lines to be included in my final list of data frames.

Can somebody help me a way to do this?

https://drive.google.com/file/d/1lJhOJadGrno1C-KZOUqxV-KNkaOHIJNk/view?usp=sharing

Answer 1

It is possible to solve this problem procedurally by thinking line-by-line or streamline-by-streamline.

For each streamline that can be matched with IntegrationTime == 0.0, extract the slice from the data frame, and append the slide to an output list (if it has more than 1 data point). A code like the following should address this problem:

import pandas as pd

# read the dataset
df = pd.read_csv("vel.csv")

# get row indexes where integrationtime is zero (start of each streamline)
start_index_list = df.loc[df['IntegrationTime']==0.0].index.values
stream_line_list = [] # the output list

for i in range(len(start_index_list)):
    # for each stream line, obtain the slice of the original dataframe that corresponds to it
    start_index = start_index_list[i]
    end_index = None
    if i+1 < len(start_index_list):
        end_index = start_index_list[i+1]-1
    stream_line_df = df.loc[start_index:end_index]
    # only append streamline with more than 1 data point
    if len(stream_line_df) > 1:
        stream_line_list.append(stream_line_df)
    
print(f"number of complete streamlines found: {len(stream_line_list)}")
```