# Understanding Confidence Interval

I am trying to understand the concept of Confidence Intervals. What is the meaning of point estimates and confidence intervals? What I understood is the point estimate in confidence interval is basically the statistics of sampling distribution. Can we say that after using central limit theorem and finding population mean using CLT rather than giving point estimate we will give confidence interval?

When you look at a linear regression $$y_i = \beta_0 + \beta_1 x_i + u_i$$, you can estimate the (ex ante unknown) coefficients $$\beta$$ using matrix algebra $$(X'X)^{-1} X'y = \hat{\beta}$$.

A point estimate would be the "best guess" $$\hat{y}=\hat{\beta} X$$.

Each $$\hat{\beta}$$ is associated with uncertainty about the estimate, expressed by the standard error of the coefficient (see this post). The confidence interval intuitively says that you want to find the range in which the true $$\hat{\beta}$$ is with 95% likelyhood (assuming a normal distribution). In this case, you can calculate a confidence interval (for $$\hat{\beta}$$) by:

$$CI_{0.95}^{\beta} = [\hat{\beta_i} - 1.96*SE(\hat{\beta_i}), \hat{\beta_i} + 1.96*SE(\hat{\beta_i})].$$

So you can say (under some assumptions) that the true value of the estimated $$\hat{\beta}$$ is within the confidence band with 95% probability. Note that in linear regression, when you say a coefficient is "statistically significant", this coincides with "the confidence band is strictly positive or negative" (does not cross zero).

Example in R:

Linear regression:

library("ISLR")
auto = ISLR::Auto
ols = lm(mpg~horsepower,data=auto)
summary(ols)


Result:

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 39.935861   0.717499   55.66   <2e-16 ***
horsepower  -0.157845   0.006446  -24.49   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.906 on 390 degrees of freedom
Multiple R-squared:  0.6059,    Adjusted R-squared:  0.6049
F-statistic: 599.7 on 1 and 390 DF,  p-value: < 2.2e-16


This tells us that when horsepower goes up by one unit, mpg goes down by -0.158 (point estimate). Now when we ask, what is the true effect (under quite some assumtions) with 95% probability, we look at the CI.

# Confodence interval
confint(ols)


Which gives:

                2.5 %     97.5 %
(Intercept) 38.525212 41.3465103
horsepower  -0.170517 -0.1451725


We can do this "manually" using:

# Get standard errors
sqrt(diag(vcov(ols)))

(Intercept)  horsepower
0.717498656 0.006445501


And we can calculate the CIs:

# Lower CI
-0.157845 - 1.96*0.006445501

# Upper CI
-0.157845 + 1.96*0.006445501


Which yields:

 -0.1704782
 -0.1452118


So we could say that the true effect of horsepower on mpg is between -0.17 and -0.15 (and since the CI does not "cross" zero, the effect is statistically significant, meaning p-value < 0.05).

• "the true value of the estimated \$\hat{\beta} is within the confidence band with 95% probability" Explicitly, this is a (Bayesian) credible interval that requires a prior distribution of the parameter.
– Dave
Oct 23 '20 at 21:23
• That‘s true, it requires assumptions about the underlying distribution Oct 23 '20 at 21:51
• Not just assumptions about the distribution from which the data were drawn, but assumptions about the distribution of the parameter.
– Dave
Oct 23 '20 at 22:05
• It would be best if you elaborate in an additional answer Oct 23 '20 at 22:31