Which analytics metrics are useful?

I am working on a home project and the question is :

A session is defined as a period of activity in the app. If there is a period longer than 5 minutes when the user does not do anything actively the session ends. The session time is the time between the first action to the last action of the session. The distribution of segments A and B is :

We want to figure out if segment A is performing different than segment B. How would do to decide if one is better than the other? Do not calculate any results but list the metrics you would calculate in the order you would calculate them.

Edit: If we assume that better is longer session times how to determine which segment is better?

First thing I would do is to try a better visualization, at least using a log scale on the Y axis.

For the question itself you want to compare the two distributions, so you could go with the usual Wilcoxon significance test (Student t-test is obviously not appropriate since the distribution is not normal). One may also be interested in the distance between the two distributions, in which case KL divergence or its variants could be used.

Knowing which one is better depends what is called "better", with this description we don't even know if it's a good thing for a user to spend more time on the app.

• That is a good approach though I also can't figure out which metric can distinguish which one is better. Jan 10, 2022 at 22:13
• @dangr figuring out which one is better is not primarily a problem of metric, it's a problem of expert knowledge. If we know what is "better", for instance longer session time, then you can do a one-sideed significance test for instance. Jan 10, 2022 at 23:16
• for longer session times the one-sided test would compare the means or medians of the 2 distributions? Jan 13, 2022 at 11:35

You want to compare the distributions, and they are not normally distributed. If log transforming the distributions makes them look roughly normal, you could compare the means of the logged data. Otherwise, comparing medians works. One of these will tell you if the groups are different on average.

You could use the common language effect size, which is the % of the time a randomly selected datapoint from group A is larger than a randomly selected datapoint in group B. That one can be quite intuitive.

You can also compare the spread of the data. standard deviation is usually the go-to, but again, data is not normally distributed so better to use interquartile range or median absolute deviation (MAD)

Additionally, as sessions time out after 5 mins, how about % of the sessions that are 5 mins? It might be useful to know if in segment A 1% of sessions time out, whereas its 5% in segment B.

• Yes, checking the sessions that have timed out is very reasonable. Jan 11, 2022 at 18:32