3
$\begingroup$

Background to the problem: I am estimating individual treatment effects using double machine learning model. I do not know true treatment effects for my problem.

Double ML: Given Y (outcome), T (treatment) and X ( features)

Y = aT + bX + error

coefficient a is of interest (measures treatment effect) .

Double ML procedure:

  1. Fit Y ~ X => Compute residuals (Y* = Y – Y’) – Residuals are treatment effects to be estimated
  2. Fit T ~ X => Compute residuals (T* = T- T’) – This model captures variation in T explained by X
  3. Fitting a model (Y* ~ T* ) on residuals will give the average treatment effects

I am fitting a linear regression model (Y* ~ T* ) and none of the coefficients are statistically significant. Instead of relying on point estimates, I am computing prediction confidence intervals and p-value to check if the predicted value is statistically significant or not.

Is this approach good?

$\endgroup$
  • $\begingroup$ Is the model overall significant? And I don’t follow how you’re proceeding with prediction confidence intervals (sounds oxymoronic) and p-values of the predicted values. Please elaborate, $\endgroup$ – Dave Jun 7 at 2:05
  • $\begingroup$ I updated my question with more details $\endgroup$ – Chandra Jun 7 at 3:26
  • $\begingroup$ Have you considered variable interactions? Are T and X correlated? Have you tried any transformations? $\endgroup$ – bstrain Jun 24 at 16:38
  • $\begingroup$ Show the full regression outcome, including all statistics $\endgroup$ – Peter Aug 23 at 21:48
0
$\begingroup$

It depends on the goal of the project.

Statistical significance is more important for publishing in academic journal. For applied projects, statistical significance is less important.

Even if a model is not statistical significant, it can have value. The model may summarize the data and identify relationships. The model maybe practically important, the effect maybe of value to practitioners.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.