0
$\begingroup$

I would like to better understand the concepts of: coefficient of variation and confidence interval.

Trivially taking the definitions from wikipedia:

  • confidence interval (CI)

    In frequentist statistics, a confidence interval (CI) is a range of estimates for an unknown parameter. A confidence interval is computed at a designated confidence level; the 95% confidence level is most common. https://en.wikipedia.org/wiki/Confidence_interval

  • coefficient of variation

    In probability theory and statistics, the coefficient of variation (CV), also known as Normalized Root-Mean-Square Deviation (NRMSD), Percent RMS, and relative standard deviation (RSD), is a standardized measure of dispersion of a probability distribution or frequency distribution. It is defined as the ratio of the standard deviation $\sigma$ to the mean $\mu$, and often expressed as a percentage ("%RSD"). https://en.wikipedia.org/wiki/Confidence_interval

But going into more detail and using for example the XGBoost library for a regression problem what are these two values and how are they obtained in practice.

EDIT 1 my progress in developing and understanding the problem so far

I start from a simpler problem the linear regression on data with Gaussian noise N(0,1)

5 different sample with same trand but different noise always Gaussian N(0,1) enter image description here Linear regression for each sample enter image description here Interval coefficient estimated with formulas $\hat{y}_h \pm t_{(1-\alpha/2, n-2)} \times \sqrt{MSE \left(\dfrac{1}{n} + \dfrac{(x_h-\bar{x})^2}{\sum(x_i-\bar{x})^2}\right)}$ enter image description here

Block boostrapping

enter image description here

I do not publish the code within the application for brevity but will upload the code to an online github repository as soon as possible For the same reason I removed the code for xgboost

$\endgroup$

1 Answer 1

1
$\begingroup$

Don't think xgbregressor is that similar to linear regression, so you might be better with understanding the latter first. You took quotes from wiki, but do you understand how it works in practice? Forget regression. Take a bunch of samples from normal distribution, get a population mean, what is the confidence interval there? How does it tie your population mean to normal distribution mean?

Next step is liner regression, and understanding how the estimator uses the data you have to give you an estimate of the statistic, and how that ties to quantities you care about.

Then you may be equipped to talk about nonlinear estimators like xgbregressor.


ADDENDUM

Not quite sure how confident you are with these. Lets look at your first comment. I think the crucial bit here is to understand that sample mean is itself a randomly distributed variable, and then question what the distribution of that variable would be. If your measurements are coming from normal distribution, then the sample mean $\bar{X}$ will be normally distributed, but it does not help you much unless you know the true variance of the distribution, which you normally do not. Sample variance $S^2$ does converge to true variance, but this is asymptotics. Luckily, for normal variables, we do have that

$$ \frac{\bar{X}-\mu}{\sqrt{S^2/n}}\sim t_{n-1} $$

For $n$ measurements. The crucial bit here is not the formula for confidence interval, but the distribution assumed by statistic of interest. Here you have the distribution for the quantity above. For fixed sample mean and sample variance the only thing that is uncertain there is the distribution mean $\mu$, so you can get the confidence interval for that. A different measurement from the same distribution will not necessarily give you a sample mean that lies in this region. For example, my first series of measurements may give me $\bar{X}_a=10$ with confidence interval $[5,15]$ and then next series may give me $\bar{X}_b=2$ with confidence interval $[-3,7]$. Distribution mean is then probably in somewhere around $5\dots 7$

In comments you mentioned bootstrapping. It will give you answers. If your sample is small it may be quite wrong, but then other approaches can be wrong too. If you have enough data, bootstrapping can produce good results, but you still need to interpret them, and for that it may be better to get a bit more familiar with simpler examples.

$\endgroup$
14
  • $\begingroup$ ok one step at a time and please for any mistakes feel free to correct me I would appreciate it. Again from wikipedia: given $X_1...X_n$ samples from normally distributed population with $\bar{X}$ sample mean and $S^2$sample variance $P(\bar{X} -c\frac{S}{\sqrt{n}}\le \mu\le \bar{X} +c\frac{S}{\sqrt{n}} ) =0.95$ After observing the sample confidential interval $\left[ \bar{x} -c\frac{S}{\sqrt{n}}\le , \bar{x} +c\frac{S}{\sqrt{n}} \right]$. This means that a new observation taken from the same population falls within the confidence interval with a 95% probability. $\endgroup$
    – Cata
    Nov 21, 2023 at 7:43
  • $\begingroup$ The estimates of the $\beta_0$ and $\beta_1$ parameters of a linear resgression are calculated from a sample of data and are therefore random variables. (econometrics-with-r.org/4.5-tsdotoe.html). Therefore, the probability of finding the true value of $\beta_i$ within the confidence interval is 95%. and the confidence interval for $/beta_i$ is defined as follows. $CI^{\beta_i}_{0.95} = [\hat{\beta_i} -1.96\times SE(\hat{\beta_i})),\hat{\beta_i} + 1.96\times SE(\hat{\beta_i}))]$(econometrics-with-r.org/5.2-cifrc.html) $\endgroup$
    – Cata
    Nov 21, 2023 at 8:27
  • $\begingroup$ prediction interval for $y_{new}$ (prediction of a future observation) $\hat{y}_h \pm t_{(1-\alpha/2, n-2)} \times \sqrt{MSE \times \left( 1+\dfrac{1}{n} + \dfrac{(x_h-\bar{x})^2}{\sum(x_i-\bar{x})^2}\right)}$ confidence interval for $\mu_Y$(prediction of the mean responce) $\hat{y}_h \pm t_{(1-\alpha/2, n-2)} \times \sqrt{MSE \left(\dfrac{1}{n} + \dfrac{(x_h-\bar{x})^2}{\sum(x_i-\bar{x})^2}\right)}$(online.stat.psu.edu/stat501/lesson/3/3.3) stats.stackexchange.com/questions/16493/… $\endgroup$
    – Cata
    Nov 21, 2023 at 10:27
  • $\begingroup$ So far I have only had a taste of the complexity of the thing even estimating how the prediction interval and confidence interval were obtained is not so easy for me without references. And we are still in the simple case of a linear regression. However, I have found methods such as bootstrapping which, but I would like to be corrected if I am wrong, would allow me to have an estimate of these confidence intervals. $\endgroup$
    – Cata
    Nov 21, 2023 at 10:40
  • 1
    $\begingroup$ ,I would like to try to explain this concept in my own words to see if I have understood it. Corrections are always very welcome. When we talk about sample mean, we have to take into consideration that we are working on a sample from the population. If we take another sample from the same population, the sample mean varies. So we can think of the sample mean as a random variable. With its own probability distribution. The t-student distribution helps us to understand how the sample mean is distributed with respect to the true value of the population mean mu. $\endgroup$
    – Cata
    Nov 23, 2023 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.