1
$\begingroup$

I have a general inference question regarding scenarios when results from data are not statistically significant but there appears to be an observable trend.

For example, treatment A and treatment B are applied to 2 independent populations. Using a ttest to analyze the resulting data (lets say the data is total revenue), the p value == .2, so the effect of treatment on revenue was not statistically significant. However, the total revenue from treatment A was observably higher in treatment B. What can I say in this regard?

I've had academic advisors recommend saying 'While the effect of treatment was not significant, a trend was observed', and then one would go on describing the trend. Is this an adequate viewpoint, or a statistical folly? What conclusions would a data scientist in industry draw and present to stakeholders from a scenario like this?

Improve this question
$\endgroup$
1
  • $\begingroup$ population has to be same and not independent to undertake a t test for correlated samples. $\endgroup$
    – Subhash C. Davar
    Jul 28, 2020 at 12:36

1 Answer 1

Reset to default
2
$\begingroup$

There is high variance within each group. Even though there is a mean difference between the groups, there is a high amount of spread within just treatment A or just treatment B.

From a statistical point of view, the difference between the groups could be due to chance because of the large spread relative to the small mean difference.

Due to the amount of randomness observed - if we did ran the experiment again, treatment B mean could be higher than treatment A mean.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.