# Statistical comparison of 2 small data sets for 2X increase in the population mean

I am trying to determine whether or not we are 90% confident that the mean of a proposed population is at least 2 times that of the mean of the incumbant population based on samples from each population which is all the data I have right now. Here are the data.

incumbantvalues = (7.3, 8.4, 8.4, 8.5, 8.7, 9.1, 9.8, 11.0, 11.1, 11.9)

proposedvalues = (17.3, 17.9, 19.2, 20.3, 20.5, 20.6, 21.1, 21.2, 21.3, 21.7)

I have no idea if either population is or will be normal.

The ratio of the sample means does exceed 2.0 but how does that translate to confidence that the proposed population mean will be at least twice that of the mean of the incumbant population with 90% confidence ?

Can re-sampling (bootstrapping with replacement) help answer this question ?

Yes, in principle, resampling can help answer this question.

incumbent <- c(7.3, 8.4, 8.4, 8.5, 8.7, 9.1, 9.8, 11.0, 11.1, 11.9)
proposed  <- c(17.3, 17.9, 19.2, 20.3, 20.5, 20.6, 21.1, 21.2, 21.3, 21.7)

set.seed(42)

M  <- 2000
rs <- double(M)

for (i in 1:M) {
rs[i] <- mean(sample(proposed, replace=T)) - 2 * mean(sample(incumbent, replace=T))
}


To make the assessment, you should choose one (not both) of the following:

A. The (two-tailed) 90% confidence interval for the difference in the (weighted) means using Hall's method is:

ci.hall <- 2 * (mean(proposed)-2*mean(incumbent)) - rev(quantile(rs,prob=c(0.05, 0.95)))
names(ci.hall) <- rev(names(ci.hall))
ci.hall

5%   95%
-0.29  2.95


This is appropriate if you have any concern about missing the possibility that mean(proposed) might actually be less than 2 * mean(incumbent).

B. The proportion of resample means >= 0 provides the (one-tailed) estimate that mean(proposed) is at least twice mean(incumbent):

sum(rs>=0)/M

 0.8915


The problem is that the samples are really rather small and resampling estimates can be unstable for small n. The same issue applies if you want to assess normality and go with parametric comparisons.

If you can get to, say, n >= 30, the approach described here should be fine.

• +1. A reference: Good, in his Resampling Methods (3rd ed., 2006, p. 19) notes that the bootstrap may be unstable for sample sizes n<100. Unfortunately, I don't have the book at hand, so I can't look up his argumentation or any references. Nov 27 '14 at 21:29
• @goangit, thank you for your reply. I don't have the software you used and don't quite follow why 0.05 and 0.95 is used in your first solution. Nonetheless, your answer inspired me to program a bootstrap which I will provide details of as "an answer". Comments on the validity of what I've done from anyone would be appreciated. Thanks Nov 27 '14 at 21:41
• @Steve, the software used here is R, it's freely available, in both senses of the word. The cutoffs used in the two tail case provide the central 90% confidence interval. Nov 27 '14 at 23:46
• @goangit, thanks. I've heard of R but can't justify the time to learn a new sw package, Mathematica is what I use, but this is not a Mathematica question. Could you explain what the (5%,-0.29) and (95%, 2.95) actually mean ? Is this saying that we are 90% confident that the ratio of the means will fall in the interval -0.29 to 2.95 ? Nov 28 '14 at 13:06

Here is what I programmed within a loop.

1. randomly take 10 values (with replacement) from the incumbant sample, determine its mean
2. randomly take 10 values (with replacement) from the proposed sample, determine its mean
3. form the ratio of the above two means and append it to a master list
4. repeat steps 1 thru 3 many times (I chose 1 million)
5. % Confidence=(number of ratios that equal or exceed 2.0/1000000)*100

Results: Exactly 897450 ratios were found to be greater than or equal to 2.0, producing a confidence of 89.745%.

Conclusion: We are less than 90% confident that the proposed population will have a mean at least twice that of the incumbant population.