Is it ok to trust regression predictions when none of the coefficients are statistically significant?

Question

Background to the problem: I am estimating individual treatment effects using double machine learning model. I do not know true treatment effects for my problem.

Double ML: Given Y (outcome), T (treatment) and X ( features)

Y = aT + bX + error

coefficient a is of interest (measures treatment effect) .

Double ML procedure:

1. Fit Y ~ X => Compute residuals (Y* = Y – Y’) – Residuals are treatment effects to be estimated
2. Fit T ~ X => Compute residuals (T* = T- T’) – This model captures variation in T explained by X
3. Fitting a model (Y* ~ T* ) on residuals will give the average treatment effects

I am fitting a linear regression model (Y* ~ T* ) and none of the coefficients are statistically significant. Instead of relying on point estimates, I am computing prediction confidence intervals and p-value to check if the predicted value is statistically significant or not.

Is this approach good?

Answer 1

It depends on the goal of the project.

Statistical significance is more important for publishing in academic journal. For applied projects, statistical significance is less important.

Even if a model is not statistical significant, it can have value. The model may summarize the data and identify relationships. The model maybe practically important, the effect maybe of value to practitioners.