I have a large dataset containing 6 characteristics (all numerical). I need to split this dataset into multiple batches to be processed in parallel, and ideally, the batches should be as equal in size as possible.
The catch is: I can only split based on the 6 characteristics, so I have to specify ranges of values for each batch. Simplified example: [1,1,1,2,2,3,3,3,3,4] will be split in two by specifying numbers 1-2 go to batch 1 and 3-4 go to batch two and I end up with [1,1,1,2,2] and [3,3,3,3,4].
I managed to make a simple division algorithm based on one characteristic, unfortunately some of the characteristics might not be suitable to split on, and I have no way to reliably predict which characteristic will create a good split.
I remember from Uni that a classification algorithm might do the trick here, but I am not sure how to implement the requirement for similar size classes.
here is the simple code I wrote for one characteristic, it may help to understand the situation:
def divide_into_groups(lst: list, num_groups=2): length = len(lst) sorted_list = sorted(lst) optimal_group_size = length // num_groups initial_group_markers = [sorted_list[i * optimal_group_size] for i in range(1, num_groups)] marker_edges = [(sorted_list.index(marker), length - 1 - sorted_list[::-1].index(marker)) for marker in initial_group_markers] final_group_markers =  for i, marker in enumerate(initial_group_markers): if i - marker_edges[i] < marker_edges[i] - i: final_group_markers.append(marker - 1) else: final_group_markers.append(marker) distances = [abs(initial_group_markers[i] - final_group_markers[i]) for i in range(len(initial_group_markers))] efficiency = (length - sum(distances)) / length print('distances from optimals: %s' % distances) print('efficiency of division is %s %%' % (efficiency * 100)) return final_group_markers
P.S. this is my first question in this forum. If I am missing some information or have made any mistake, please comment and I will fix as soon as possible.