0
$\begingroup$

Let's consider we have two datasets. Dataset "A" and Dataset "B".

Dataset "A" has two columns. Supplier_id and "Status" (pass and fail are values for status columns)

Dataset "B" has 100 plus columns which in a way kind of have some longitudinal data (multiple time points for same supplier) about the suppliers. For ex, it has revenue from each order, product type etc

These two datasets are linked through a identifier column.

However, our objective is to find out whether there is any pattern that we can uncover from Dataset B based on the "Status" column from Dataset "A".

For ex: I want to know whether high revenue suppliers often "Pass" the test or "Low" revenue suppliers often "Pass" the test..

So, I would like to do this sort of comparison for 100 plus columns in Dataset "B" against the "Status" column.

a) How can I do this efficiently for 100 plus columns (to compare with Status column)

b) Since, I know the label (Status), do you think I can hide it and do the clustering on "Dataset B". Once I find "N" number of clusters, I can then investigate them to see how many records have "Pass" or "fail" status?

c) For question a), Is there any python package that can help me do this for 100 plus columns and provide us some statistics that can help us uncover something new

$\endgroup$

1 Answer 1

1
$\begingroup$

Maybe Left Merge. Just a thought.

pd_merge = pd.merge(A, B, on='ID', how='left)

Then, when you have your merged data frame, set your target variable and do a feature importance exercise, because you already know the target variable ('Status')

More info here.

https://towardsdatascience.com/why-and-how-to-use-merge-with-pandas-in-python-548600f7e738

Also, here.

https://mljar.com/blog/feature-importance-in-random-forest/

$\endgroup$
1
  • $\begingroup$ thanks for the help. upvoted. but how does computing feature importance, would help in finding the pattern? I would like to find answers to questions like "whether high revenue customer is mostly attaining pass status or whether it is low revenue customer that is assigning pass status? $\endgroup$
    – The Great
    Apr 6, 2022 at 5:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.