1
$\begingroup$

I am trying to perform two sample t test. My data set consists of 744 rows and 186 columns for which I have calculated total sum and mean. I need to perform two sample t test. My csv looks like this from which I have to calculate ttest and rank sum test for each row as individual row denotes separate ID and have the corresponding values :

SRA ID  ERR169499            ERR169498           ERR169497
Label   1                    0                   1
TaxID   PRJEB3251_ERR169499  PRJEB3251_ERR169499 PRJEB3251_ERR169499
333046  0.05                 0.99                99.61
1049    0.03                 2.34                34.33
337090  0.01                 9.78                23.22

The labels 0 and 1 are for case and control respectively. So far I have done this:

import pandas as pd
import numpy as np
from scipy.stats import ttest_ind
from scipy.stats import ranksums

def transposer(filename):
file = open(filename, 'rt')
pd.read_csv(file).T.to_csv(str(filename).split("/")
[-1].split(".")[0]+'_transposed.csv',header=False)


pd.read_csv('project.csv').T.to_csv('transposed.csv', header=False)

file = open('transposed.csv', 'rt')
out = open('final_out.csv', 'w')
meta = open('Meta3251.csv', 'rt')
contents = {}
for ids in meta:
    contents[ids.split(',')[1]]=ids.split(',')[-1]
count = 0
for row in file:
    if count == 0:
    out.write('SraID, Label,'+row)
    count=1
else:

    try:
        pid = row.split(',')[0].split('_')[1]
out.write(pid.replace('\n','')+','+contents[pid].replace('\n','')
+','+str(row))
        out.flush()
    except:
        print(pid)
        pass
file.close()
out.close()
transposer('final_out.csv')
file1 = open('final_out_transposed.csv','rt')
label = []
data = {}

x = open('final_out_transposed.csv','rt')
for r in x:
    datas = r.split(',')
    if datas[0] == ' Label':
        label.append(r.split(",")[1:])
label = label[0]
label[-1] = label[-1].replace('\n','')
counter = len(label)
for row in file1:
    content = row.split(',')
if content[0]=='SraID' or content[0]== 'TaxID' or content[0]==' Label':
    pass
else:
    dt = row.split(',')
    dt[-1] = dt[-1].replace('\n','')

    data[dt[0]]=dt[1:]
keys = list(data)
sum_file = open('sum.csv','w')
sum_file.write('TaxId,sum_case,sum_ctrl,case_count,
ctrl_count,case_mean,ctrl_mean,\n')
for key in keys:
    sum_case = 0
    sum_ctrl = 0
    count_case = 0
    count_ctrl = 0
    mean_case = 0
    mean_ctrl = 0
for i in range(counter):
    if label[i] == '0':
        sum_case=np.float64(sum_case)+np.float64(data[key][i])
        count_case = count_case+1
        mean_case = sum_case/count_case
    else:
        sum_ctrl = np.float64(sum_ctrl)+np.float64(data[key][i])
        count_ctrl = count_ctrl+1
        mean_ctrl = sum_ctrl/count_ctrl
sum_file.write(key+','+str(np.float64((sum_case)))+','

+str(np.float64((sum_ctrl)))+','+str(np.float64((count_case)))        
+','+str(np.float64((count_ctrl)))+','+str(np.float64((mean_case)))
+','+str(np.float64((mean_ctrl)))+'\n')
sum_file.flush()
sum_file.close()

df  = pd.read_csv('final_out_transposed.csv', header=[1,2], index_col=[0])
case = df.xs('0', axis=1, level=0).dropna()
ctrl = df.xs('1', axis=1, level=0).dropna()
(tt_val, p_ttest) = ttest_ind(case, ctrl, equal_var=False)
print (tt_val)
print (p_ttest)

I am getting the error:

ValueError: operands could not be broadcast together with shapes (92,) (95,)

How can I handle this error. I cannot change my data.

$\endgroup$
  • $\begingroup$ would you say the error is for which line? $\endgroup$ – Media Jan 31 '18 at 14:43
  • $\begingroup$ It is in the line (tt_val, p_ttest) = ttest_ind(case, ctrl) $\endgroup$ – K.S Jan 31 '18 at 14:54
  • $\begingroup$ When you drop rows based on na's and assign it two variables, they might not be of same length. What you need to do inplace the dropna's and then take ctrl from there. $\endgroup$ – Kiritee Gak Jan 31 '18 at 15:06
  • $\begingroup$ @KiriteeGak eveen with doing that im still having the error. The columns for case and control are actually 92 and 95. $\endgroup$ – K.S Jan 31 '18 at 16:25
  • $\begingroup$ @Media any help on how to numpy broadcast will solve this I guess $\endgroup$ – K.S Jan 31 '18 at 16:27
2
$\begingroup$

The answer to this question would be : The objects created by the xs method of the Pandas DataFrame look like two-dimensional arrays. These must be flattened to look like one-dimensional arrays when passed to ttest_ind. The values attribute of the Pandas objects gives a numpy array, and the ravel() method flattens the array to one-dimension. It would go like :

df  = pd.read_csv('final_out_transposed.csv', header=[1,2], index_col=[0])
case = df.xs('0', axis=1, level=0).dropna()
ctrl = df.xs('1', axis=1, level=0).dropna()
(tt_val,p_ttest ) = ttest_ind(case.values.ravel(), ctrl.values.ravel(), 
equal_var=False)
print (tt_val)
print(p_ttest)
| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Congrats for your effort $\endgroup$ – Media Feb 1 '18 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.