0
$\begingroup$

I have two data frames src, dst for a transfer learning task in bioinformatics, one has labels, the other does not. What I want to do is make sure that outside of the labeling and ID columns they share the same features in the same order for putting into numpy.

Specifically:

the columns for src are ['PATIENT_ID','SAMPLE_ID', <buncha_stuff1>, "SUBTYPE","SUBTYPE_SHORT","LABEL"]

and ['PATIENT_ID','SAMPLE_ID', <buncha_stuff2>] for dst.

The buncha stuff is not the same in both but there is a large intersection. My goal is to take buncha_stuff = buncha_stuff1.intersection(buncha_stuff2) and use that as the non-label and non-id columns in src and dst. My approach is

meta_cols ={"PATIENT_ID","SAMPLE_ID","SUBTYPE","SUBTYPE_SHORT","LABEL"}
common_genes= list(set(src.columns).intersection(set(dst.columns))-meta_cols)
src_cols = ["PATIENT_ID","SAMPLE_ID"] + common_genes + ["SUBTYPE","SUBTYPE_SHORT","LABEL"]
dst_cols = ["PATIENT_ID","SAMPLE_ID"] + common_genes
src = src[src_cols]
dst = dst[dst_cols]

At the beginning src has 18605 columns and dst has 17666. The listsrc_cols has length 17651 and dst_cols has length 17648, however when I check the data frame shapes after doing

src = src[src_cols]
dst = dst[dst_cols]

src has 17672 columns and dst has 17666.

it seems like pandas is just adding in columns and I can't find them? what is going on? I'm going insane

$\endgroup$
0

2 Answers 2

1
$\begingroup$

It could be that some of the columns names are duplicated. Using src[src_cols] would return any column with a column name in the src_cols list even if they were duplicates.

A quick check is: len(set(src.columns)) - len(src.columns). If this is !=0 then there must be duplicates. Hopefully it will return the number of additional columns you have.

$\endgroup$
1
$\begingroup$

Your approach makes total sense to me however, you could use another approach to verify what's going on:

meta_cols =["PATIENT_ID","SAMPLE_ID","SUBTYPE","SUBTYPE_SHORT","LABEL"]
src_cols = src.columns
dst_cols = dst.columns
common_genes= list(set(set(src_cols) and set(dst_cols)))
common_genes = [x for x in common_genes if x not in meta_cols]
src_cols = ["PATIENT_ID","SAMPLE_ID"] + common_genes + ["SUBTYPE","SUBTYPE_SHORT","LABEL"]
dst_cols = ["PATIENT_ID","SAMPLE_ID"] + common_genes
src = src[src_cols]
dst = dst[dst_cols]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.