Pandas datastructure

Question

I'm trying to analyze database performance over a period of time and detect anomalies. The database server consists of many threads that perform different actions. I run a query to determine the number of active threads and the action they are performing.

A sample dataset is below:

My Objective: I need to analyze over a period of time and determine what is normal at a given timestamp and detect any abnormalities. For example, Monday at 10 am, there are 10 active threads; out of which, there are 2 threads with the action 'Preparestatement' and 10 threads with the action 'Readtable'. Any other thread actions is potentially an anomaly.

As you can see from the image above, the actions (executestatement, Fetchcursor and so on) could be different at each timestamps. I want to understand if the structure of the pandas dataframe is the right choice to meet my objective.

Answer 1

Not sure if I understood your question correctly, but to dodata analysis, you would need to have a proper DF.

You have something like:

time A B C
1    3 4 6 
time B C D
2    9 4 6

You need one table header, so remove any other header text in the rows > 1.

Also (of course) the content of each row must match the column it belongs to.

So the DF above would change to:

time A B C D
1    3 4 6 0
2    9 4 0 6

Note the "zero" entries in time=1 -> D=0 and time=2 -> C=0. Here the "zero" is chosen to fill the gaps (zero activity in this case).

This type of data representation is what you usually use for any kind of modelling.