What kinds of ML models should I use when the outcome variable does not vary with time but only vary across individuals and groups?

I am trying to predict individuals’ income in 2018 using 18 years worth of data for people who were born in 1978,1979, and 1980 on many variables such as family income, location, family members’ education etc.(there are at least 50 variables like that) The goal is to see which stage of the childhood better predicts income in the labor market at age 38. As one can infer from the problem, my outcome variable (income at age 38) does not vary with time but vary across individuals as well as birth cohorts and the covariates may or may not vary with time, birth cohorts, and across individuals. I believe I can not use panel models. How should I think about modeling this?

A small sample dataset looks like this : The dependent variable is labinc38, which is labor income of an individual at the age of 38.

id year sex birthyear totfaminc occup_h car housevalue labinc38
41030 1978 Female 1978 12003 962 Yes 19000 57800
41030 1979 Female 1978 13946 962 Yes 11000 57800
41030 1980 Female 1978 17190 283 Yes 9500 57800
41030 1981 Female 1978 26200 282 Yes 9000 57800
16176 1979 Male 1979 14176 0 Yes 0 36000
16176 1980 Male 1979 16000 0 Yes 0 36000
16176 1981 Male 1979 13605 694 Yes 0 36000
16030 1980 Female 1980 16000 0 Yes 0 30000
16030 1981 Female 1980 13605 694 Yes 0 30000

Should I covert my data to wide format as then model it as though it is cross-sectional?

If you have many different groups with different objectives, you could predict their variations instead of their row values.

For instance:

• If group A is earning 50000USD in 1979 and 53500USD in 1980, there is a 7% increase.
• If group B is earning 80000USD in 1979 and 84000USD in 1980, there is a 5% increase.

Consequently, group A performs better than group B, even if the raw value is smaller.

In this way, every group could be compared to each other, but it should be applied to the relevant features only (ex: distance from work might not be relevant).

• Hello @Nicholas, thank you so much for your suggestion. I do appreciate it. How about if I convert the data from long to wide and then build the model as if I am dealing with the cross sectional data? Nov 8, 2022 at 16:38
• Could you give a short and simple example editing your question? Nov 8, 2022 at 16:40
• Yes, I have shared a subset of the data. Thank you! Nov 8, 2022 at 17:41

The data provided in the question is limited in quantity and in explanatory variables. You presumably have at the least much more data to work with - otherwise there is no predictive power.

From what you showed the following are explanatory variables:

year
sexbirthyear
totfaminc
occup_h
car
housevalue


Your label is labinc38

This is a simple case of regression . Which model to use will depend on the form of the curve that best describes the labinc38 given the explanatory variables. With the very little data shown it is not possible to discern what that form would be. Let's assume you do have much more data and that they have at least a few tens of observations . Then you can start to examine whether there are approximately linear relationships of the income at age 38 with one or more of those variables. If the relationships are not linear then explore other relationships including polynomial (quadratic/cubic etc), logarithmic, exponential ,etc. The next consideration are product terms : ie. non-linear combinations of the explanatory variables. Even more complicated relationships can occur: but then you start to run into the downsides of:

• over-fitting
• lack of explainability of your model. I.e you would not be able to verbalize the effects of each variable on the results.

So it would be best to start out simple and try to keep it as simple as possible.

• Thank you so much @WestCoastProjects! I do have more data. This is just a small subset of it to show how the data looks like. The explanatory variables have values that change across individuals and over time. But the target variable does not change with the time as you pointed out it is the income at 38. So the panel models won’t be useful here. I will start with the regression part as you suggested. Thanks a lot again! I appreciate the help. Nov 9, 2022 at 0:49