note in advance: I'm new to data-analysis and although my major civil engineering taught me about statistics, I did not apply it the way I would have encountered in real life, or in this field of work. I don't have any person in my circle of contacts that I can get advice from, that's why I'm asking here.
the description:
I have a dataset of a large group of people nearly 200, and that I'm trying to arrange these people into group, (each group is from 12 to 14 person). the dataset contains parameters of birthyear and home-state all numerical.
a previous code that I worked on achieved that be just shuffle them into groups. but the problem was that: (one group could contain people mostly from one state, and other groups could be non-representative of that state). and I don't actually like how it worked, as I would describe it "lazy grouping"
question 1: so, If I have a the large group, how can I split it into subgroups with the same properties or near enough the large group??
here are some histograms that show my case:
something I'm thinking of:
although the large group can give me the mean/var/std, but because I'm subtracting from it to make subgroups, this means that I should not lookup for ideal group. and that my solution should be to fix uniform shuffling. so I searched and saw that numpy can offer weighted shuffling. but it was not as quite as I expected. and by so I need a to make a shuffling function for my purpose.
the algorithm that should work for me:
# df = 'MY_ORIGINAL_DF'
home_state = df.home_state
birthyears = df.birthyear
home_state_count = home_state.value_counts()
home_state_normalized = home_state_count / home_state_count.sum()
# +-----+-----+-----+-----+
# | 1 | 2 | 3 | 4 | corresponding home_state
# home_state_normalized = +-----+-----+-----+-----+
# | 0.1 | 0.4 | 0.2 | 0.2 | its propbablity
# +-----+-----+-----+-----+
birthyears_count = birthyears.value_counts()
birthyears_normalized = birthyears_count / birthyears_count.sum()
# +------+------+------+
# | 1970 | 1980 | 1990 | corresponding birthyears
# birthyears_normalized = +------+------+------+
# | 0.60 | 0.10 | 0.30 | its propbablity
# +------+------+------+
## calculating weighted_matrix:
# +-----+ +------+------+------+
# | 0.1 | | 0.06 | 0.01 | 0.03 |
# +-----+ +------+------+------+
# | 0.4 | +------+------+------+ | 0.24 | 0.04 | 0.12 |
# weighterd_matrix = +-----+ (outer_product) | 0.60 | 0.10 | 0.30 | = +------+------+------+
# | 0.2 | +------+------+------+ | 0.12 | 0.02 | 0.06 |
# +-----+ +------+------+------+
# | 0.2 | | 0.12 | 0.02 | 0.06 |
# +-----+ +------+------+------+
now each cell of weighterd_matrix corresponds to the pair (birthyear, home_state). (and to be clear: I know how to apply matrix multiplication in python, it's not a problem, but I made it look visually easy to read)
question 2: how can I apply shuffling to my dataframe based on this weighted matrix??
things you can help me with:
- suggest topics I can search about.
- suggest how I can continue with my implementation of
weighted_matrix. - suggest docs that can guide me.
- be positive. I'm a messy-thought person and I tried so much to organize me problem and my description
