2
$\begingroup$

I have a DataFrame that pairs one or more labels to a sample group and id, for a given sample stored in a database at SampleGroup/SampleID:

enter image description here

There are ~100 labels. I want to create binary models to do classification on each label, and then run these models in parallel to do multi-class classification. To store these models, I am creating a dictionary of form

{label_1:[df_1, model_object_1],
 label_2:[df_2, model_object_2],
...,
label_n:[df_n, model_object_n]
}

Where each df is a DataFrame of the form above, except that the value of the 'Labels' column is replaced with a 1 or 0, depending on whether dictionary key 'label_i' is in the original label list for that row. Here's the code that (should) do that, that has been giving me some trouble:

models = dict.fromkeys(target_labels, [])

for label in target_labels:
    label_list = []
    for multi_label_list in df['Labels']:
        if label in multi_label_list:
            label_list.append(1)
        else:
            label_list.append(0)

    data = {
        'SampleGroup':df['SampleGroup'].copy(),
        'SampleID':df['SampleID'].copy(), 
        'Labels':label_list
    }

    models[label].append(pd.DataFrame(data=data, index=df.index))
    print(len(models[label]))

When I run this, each new binary label_list that is created for a label gets appended to every model in the dictionary, as if I'm creating a reference to the same label_list (similar to how df2 = df would create a reference to df, instead of a copy). The output of the above code tells the story clearly:

[len(models[label]) increases by 1 with each iteration of append.[2]

I managed to hack a fix for this by assigning each new DataFrame to the key instead of appending it to the key's value list:

models[label] = (pd.DataFrame(data=data, index=df.index))

What property of DataFrames (or perhaps native Python) am I invoking that would cause this to work fine, but appending to a list to act strangely?

$\endgroup$
2
$\begingroup$

The problem is when you create the dictionary models using models = dict.fromkeys(target_labels, []), you actaully only created one empty list, and all keys point to that list. Everything you append goes into that same list.

For example,

models = dict.fromkeys('abcd', [])
print(models)
print(models['a'] is models['b'])
models['a'].append(3)
print(models)

will return

{'d': [], 'a': [], 'b': [], 'c': []}
True
{'d': [3], 'a': [3], 'b': [3], 'c': [3]}

Notice that models['a'] is models['b'] is True.

You can create the dictionary instead using

models = dict([(key, []) for key in 'abcd'])
print(models)
print(models['a'] is models['b'])
models['a'].append(3)
print(models)

Now it returns

{'d': [], 'a': [], 'b': [], 'c': []}
False
{'d': [], 'a': [3], 'b': [], 'c': []}

Notice that models['a'] is not models['b'].

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.