How to do modelling for pairs of non i.i.d. data?

I have a dataset in which i have the labels for candidates on whether they would be hired,interviewed_and_failed,not_interviewed_at_all. The task is to predict for new jobs/new candidates what these decisions would be.

Assume that I have some feature space $$x_c$$ for each candidate and $$x_j$$ for each job. A simple model would task be $$y = f(x_c,x_j)$$ where $$y$$ represents the hiring decision and $$f$$ represents some sort of machine learning model.

The question is how should i split and train the data because the data is not i.i.d.

1. If i just split the model randomly it might not generalize to new jobs/new candidates which is ultimately what the model should do. i.e. $$x_c$$ and $$x_j$$ is present in both training and validation set.

2. Ignoring the validation aspect how do i prevent the model from memorizing the jobs/candidates that it has seen. This is because some jobs such as cashier has many successful hires while a CTO has only 1 hire. So the model just learns the acceptance ratio for each job rather than learning how to rank the candidates for each job.

is it possible to have a model that does something like $$y = f(x_c|x_j)$$? And how do I train such a model?

• Could you please add a minimum reproducible example of what your data look like? Do you have the same candidate for n different jobs? Commented Mar 2, 2022 at 22:26

I would use leave one out, where the partition is on a per-job basis.

• Completely isolate a particular job for testing
• Divide your remaining data into training and validation data as you see fit.
• train/optimize your model with the train and validate data
• test on the lone, reserved job.
• repeat for another job, until all jobs have been tested.

Do this for every job, so you'll train a new model for all the data minus 1 of your jobs, and you'll have tested each of your jobs. Then, you can use the aggregate results to approximate how your model will generalize to a new job being introduced. Your final model can be trained on all data, be a random model from the batch, an ensemble of all the models trained; it's up to you, all have their costs and benifits.

If your dataset is too large for this "wasteful" strategy to be practical, you can use k-fold to achieve a similar result, but doing it with multiple jobs at a time. If that, still, remains too wasteful, you can do your 70/10/20 train/validate/test split on a per-job basis, not on an individual datum basis.

As for reducing the risk of memorization, that's a feature engineering question which is highly dependent on your particular problem. A good solution might be to include certain features in your hyperparameter tuning strategy, such that you can iteratively learn which features seem useful, and which don't. If you find yourself constantly worrying about memorization, you probably don't have enough data and/or you have too many features.

In order to have an appropriate model validation, you should always think of how the inference/prediction phase will work once your model is in production.

In this case, if I did not misunderstand, you want to make predictions for a candidate $$c_i$$ on a set of different jobs $$j_k$$ for $$k$$ in $$[{1,...,n}]$$ for a multi-class output $$y_i$$ in $$S$$ where $$s_i$$ is one of the possible statuses of the candidate after the interview.

So one way to correctly evaluate your model's performance would be by making not random splits by Grouped splits.

From Scikit-learn documentation:

GroupKFold is a variation of k-fold which ensures that the same group is not represented in both testing and training sets. For example, if the data is obtained from different subjects with several samples per subject and if the model is flexible enough to learn from highly person-specific features it could fail to generalize to new subjects. GroupKFold makes it possible to detect this kind of overfitting situations.

Imagine you have three subjects, each with an associated number from 1 to 3:

from sklearn.model_selection import GroupKFold

X = [0.1, 0.2, 2.2, 2.4, 2.3, 4.55, 5.8, 8.8, 9, 10]
y = ["a", "b", "b", "b", "c", "c", "c", "d", "d", "d"]
groups = [1, 1, 1, 2, 2, 2, 3, 3, 3, 3]

gkf = GroupKFold(n_splits=3)
for train, test in gkf.split(X, y, groups=groups):
print("%s %s" % (train, test))


In your case, your groups would be the different jobs (cashier, attorney, etc) So you would be training in a group of candidates that belong to a group of jobs and testing on a different set of jobs that are not present in the training stage.

That could give you a better idea of your model's ability to correctly classify candidates whose jobs had not been seen.

The problem you describe could be framed as multiclass classification. Given a set of features predict which of one of three discrete outcomes (i.e., hired, interviewed_and_failed, not_interviewed_at_all) is most likely.

It is best practice to do stratified splits between train, validation, and test date. Stratified splitting attempts to maintain the overall proportions of each class during each split.

Your second question is about failing to generalize. That is a separate and relatively complex question. The most common general advice is to collect a lot of high quality data and train a model that does well both on training data and unseen, hold-out data.