2
$\begingroup$

For each of the following sample sizes [3,5,7,9], I Need to calculate the 25th percentile of the standard deviation for the values sampled. And the total number of trials will be 10.

DataFrame

US    China  Korea   India   UK
196    213    9        77    84
105    122   16        52    60
346    305  -12        78    69
155    113  -27        42    30
210    200  -5         68    65
212    190  -10        70    62
227    219  -4         90    87
106     96  -9         89    81
367    326  -11        91    80
86     104   21        69    83
200    194   -3        77    75

My code:

    sample_sizes = [5, 7, 9]
    num_trials = 10
    col_index = 3
    p = 25
    df=pd.read_csv(filename)
    std_list=[]
    for i in sample_sizes:
        for j in range(num_trials):
            Sample=df.sample(n=i, random_state=j)
            col=Sample.iloc[:,col_index].std()
            std_list.append(col)
        ptile=np.percentile(std_list,p)
        print(ptile)  

The code above gives the following ptile values:

   9.079918996054708
   9.408029717257989
   11.572161922408418

However, if I do remove the first loop and hardcode the value for sample size the ptile value changes.

    num_trials = 10
    col_index = 3
    p = 25
    df=pd.read_csv(filename)
    std_list=[]
    for j in range(num_trials):
        Sample=df.sample(n=7, random_state=j)
        col=Sample.iloc[:,col_index].std()
        std_list.append(col)
    ptile=np.percentile(std_list,p)
    print(ptile)  
The code above gives `ptile = 11.878249130348483` for sample size 7 
whereas the first code gives `ptile = 9.408029717257989` for a sample size of 7.

I will greatly appreciate it if someone can explain the reason for this inconsistency.

$\endgroup$
0
$\begingroup$

This happens because you forget to reset std_list after starting with a new sample size. With the following change to the first version of your code (the one with varying $i$), you get the desired results:

sample_sizes = [5, 7, 9]
num_trials = 10
col_index = 3
p = 25
df=pd.read_csv(filename)
for i in sample_sizes:
    std_list=[]
    for j in range(num_trials):
        Sample=df.sample(n=i, random_state=j)
        col=Sample.iloc[:,col_index].std()
        std_list.append(col)
    ptile=np.percentile(std_list,p)
    print(ptile)  
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.