Your treatment group and control group are different in an unknown number of ways. There are a number of covariates that make these groups different. They were initially randomly selected so the difference between any static covariates (age, demographics) won't be statistically significant (unless you're unlucky). However their past exposure to different experiences with your company, and any resulting effects of that, are significant differences between these groups. Directly measuring and controlling for these differences (stratifying by them, adding them to a model, etc.) would help achieve ignorability (that all differences in covariates between the treatment + control groups are accounted for).
Difference in difference is a way to estimate the experiment's effect for a treatment and control group that have different initial values of the key metric, so they are different in at least some way. It accounts for the pre-treatment difference between the control group and the treatment group.
In the common AB test, your treatment effect is the post-treatment difference between treatment and control. But with difference in difference, we subtract from this effect the pre-treatment difference between the treatment and control.
Another way to think of it is that we take the treatment + control group's post-treatment values and subtract their respective pre-treatment values, before the final comparison.
The comparisons are of course a statistical test on the difference of these modified final values, hence the "difference in difference." You can also take a bayesian testing approach to get a probability of one being superior to the other rather than the frequentist question of statistical significance.
However, the following assumption will still apply to either test:
Although difference in difference is a way of handling the pre-treatment difference, it assumes that the post-treatment difference should've been the same, if there had not been a treatment. Put another way, the key assumption in difference in difference is that the treatment group would have undergone the same CHANGE in its key metric as did the control group, if the treatment group had not received the treatment. We are assuming that the pre-treatment difference between the treatment + control groups should have been equal to the post-treatment difference between the treatment + control groups if no treatment was applied. The difference between this hypothetical (counterfactual) and the true outcome is assumed to be our effect size.
So, we need to evaluate this assumption in our context. Do we think that your treatment group would have undergone the same change in its metric that the control group underwent? Without knowing much about the covariates that make those groups different, it's hard to say that with any confidence. But to be fair, assumptions like these are commonplace in causal inference.
In the end, you have to make a business decision about whether you have the time to re-run an A/B, and whether you shouldn't take action in the meantime. Difference in difference will at least let you account for the initial difference between treatment and control, but you'll have to think about the chances that these two groups would have made the same change over this time. If it's a short time and a stable metric, then maybe it's not such a bad assumption.
If the control group's metric didn't change over the time period, then you're just looking at a pre vs. post on the treatment group. From that perspective, your measure is more like the Average Treatment effect on the Treated (ATT), rather than the Average Treatment Effect (ATE) on all users. That is to say that your experiment will tell you how users like the even id users react to the change, but it won't tell you how users like the odd users would react to the change.
Perhaps you can dive into the differences between even and odd users (e.g. purchasing behavior, etc.), to better understand what you've learned.
