The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. I am confused about the first step for doing this because there are many datasets (4). If the first step is to preprocess the data, which dataset will be preprocessed first? I am using the R language.
Thank you.
I am assuming this is competition data?
Basically, start with the previous_data and information_history data which seemingly provide extra information for observations in the training and testing datasets. You are probably given lookup ids that allow you to relate any particular row of these extra datasets to that of the training/testing datasets.
Most people will aggregate these extra variables so that one row = one unique lookup id (i.e., aggregate these datasets by id) by using various sample statistics; means, maxes, mins, variances, kurtosis, percentiles (like the median), etc. I've even seen people attach the target variable from the training dataset on to the "extra" datasets so that they can run regressions/classifiers on these datasets to get more features that give better aggregations. One extra suggestion: information_history and previous_data sounds like there is a time element. Perhaps be mindful of using more recent data, perhaps having features like "x mean in past k time periods", or "y median in n most recent loans", etc. I don't know what the datasets are but that is just a suggestion.
After that is done, then simply join these aggregated summary statistics to the training/testing data sets by the lookup = id column. In R, this is quite easy to do with the tidyverse and left_join/right_join, for example.
I hope this answers your question.
