I am collecting metrics on 6 REST services from a microservices architecture. For each instant collected, I extract two CSV from each service. One CSV contains three latency metrics (99th quantile, 50th quantile, average). And another CSV has the number of responses per second that the service returned with the HTTP codes 200 and 400.

Examples of each of the CSVs:


"Time", "99th quantile", "50th quantile", "Mean", "IsError"
2023-02-23 15:20:30,2.45,0.577,0.602,True
2023-02-23 15:21:00,0.939,0.424,0.457,True
2023-02-23 15:21:30,0.740,0.417,0.456,True
2023-02-23 15:22:00,0.965,0.396,0.443,True
2023-02-23 15:22:30,2.34,0.438,0.547,True


2023-02-22 21:18:30,216,0,False
2023-02-22 21:19:00,280,0,False
2023-02-22 21:19:30,242,0,False
2023-02-22 21:20:00,311,0,False

In addition to the metrics, it has a column with the Time and a column with the label whether it is a boolean value.

The CSV file names always start with the service name and have the word "latency" in the case of latency metrics and "QPS" in the case of wanted per second.


Cart latency-data-as-seriestocolumns-2023-02-23 15_53_38.csv
Cart QPS-data-2023-02-23 15_53_26.csv
Catalogue latency-data-as-seriestocolumns-2023-02-23 15_53_20.csv
Catalogue QPS-data-2023-02-23 15_53_13.csv
Frontend latency-data-as-seriestocolumns-2023-02-23 15_54_54.csv
Frontend QPS-data-as-seriestocolumns-2023-02-23 15_54_48.csv
Orders latency-data-as-seriestocolumns-2023-02-23 15_53_54.csv
Orders QPS-data-2023-02-23 15_53_47.csv
Payment latency-data-as-seriestocolumns-2023-02-23 15_54_10.csv
Payment QPS-data-2023-02-23 15_54_00.csv
Shipping latency-data-as-seriestocolumns-2023-02-23 15_54_24.csv
Shipping QPS-data-2023-02-23 15_54_17.csv
User latency-data-as-seriestocolumns-2023-02-23 15_54_40.csv
User QPS-data-as-seriestocolumns-2023-02-23 15_54_32.csv

I wanted to make a dataset where it reads all the CSV from a paste and creates the whole dataset for training and validation.

In the end, I would have a dataset with the following format:

"Time","99th quantile","50th quantile","Mean","2xx","4xx/5xx","IsError","Service"
2023-02-06 16:13:00,0.0970,0.00402,0.00771,254,0,True,Orders
2023-02-06 16:13:30,0.0700,0.00377,0.00614,267,0,True,Orders
2023-02-06 16:14:00,0.0208,0.00328,0.00388,251,0,True,Orders
2023-02-06 16:14:30,0.0971,0.00349,0.00655,273,0,True,Orders
2023-02-06 16:15:00,0.0232,0.00323,0.00443,276,0,True,Orders
2023-02-06 16:15:30,0.00995,0.00309,0.00380,69,0,True,Orders
2023-02-06 16:16:00,0.00957,0.00283,0.00316,171,0,True,Orders

Is it possible to put all this information together in a single DataFrame? Knowing that a collected moment is represented by two CSVs and that in the same folder, I will have several collection time periods.

  • 1
    $\begingroup$ Sounds fun! Yes, after loading all the files into their own dataframes (however you think is best), it sounds like you can merge on the Time column in all dataframes in pandas using merge, join, or concat (again, depending on what you think is best). $\endgroup$
    – m13op22
    Mar 3 at 17:45


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.