0
$\begingroup$

I am working on a home project and the question is :

A session is defined as a period of activity in the app. If there is a period longer than 5 minutes when the user does not do anything actively the session ends. The session time is the time between the first action to the last action of the session. The distribution of segments A and B is : enter image description here

We want to figure out if segment A is performing different than segment B. How would do to decide if one is better than the other? Do not calculate any results but list the metrics you would calculate in the order you would calculate them.

Edit: If we assume that better is longer session times how to determine which segment is better?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

First thing I would do is to try a better visualization, at least using a log scale on the Y axis.

For the question itself you want to compare the two distributions, so you could go with the usual Wilcoxon significance test (Student t-test is obviously not appropriate since the distribution is not normal). One may also be interested in the distance between the two distributions, in which case KL divergence or its variants could be used.

Knowing which one is better depends what is called "better", with this description we don't even know if it's a good thing for a user to spend more time on the app.

Improve this answer
$\endgroup$
3
  • $\begingroup$ That is a good approach though I also can't figure out which metric can distinguish which one is better. $\endgroup$
    – alan watt
    Jan 10, 2022 at 22:13
  • 1
    $\begingroup$ @dangr figuring out which one is better is not primarily a problem of metric, it's a problem of expert knowledge. If we know what is "better", for instance longer session time, then you can do a one-sideed significance test for instance. $\endgroup$
    – Erwan
    Jan 10, 2022 at 23:16
  • $\begingroup$ for longer session times the one-sided test would compare the means or medians of the 2 distributions? $\endgroup$
    – alan watt
    Jan 13, 2022 at 11:35
1
$\begingroup$

You want to compare the distributions, and they are not normally distributed. If log transforming the distributions makes them look roughly normal, you could compare the means of the logged data. Otherwise, comparing medians works. One of these will tell you if the groups are different on average.

You could use the common language effect size, which is the % of the time a randomly selected datapoint from group A is larger than a randomly selected datapoint in group B. That one can be quite intuitive.

You can also compare the spread of the data. standard deviation is usually the go-to, but again, data is not normally distributed so better to use interquartile range or median absolute deviation (MAD)

Additionally, as sessions time out after 5 mins, how about % of the sessions that are 5 mins? It might be useful to know if in segment A 1% of sessions time out, whereas its 5% in segment B.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes, checking the sessions that have timed out is very reasonable. $\endgroup$
    – alan watt
    Jan 11, 2022 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.