I have population data from Census.gov:

Total US population by age by year from 1940 through 2010

Depending on the range of decades, the data is missing discrete population values for ages greater than a certain age. Instead an aggregate amount is provided that represents all ages greater than the cutoff.

Specifically it follows this pattern:

  • 1940 to 1979: Discrete data from 0 to 84 and aggregate for ages 85 and greater
  • 1980 to 1999: Discrete data from 0 to 99 and aggregate for ages 100 and greater
  • 2000 to 2010: Discrete data from 0 to 84 and aggregate for ages 85 and greater

The desired outcome is to have discrete data points for each age and year from 0-99 and then an aggregated lump sum figure for ages 100 and greater.

Therefore I want to input the missing discrete population values for ages 85 to 100 for years 1940 through 1979 and years 2000 through 2010.

And I want to use the actual discrete population values for ages 85 to 100 for years 1980 through 1989 to achieve that outcome.

Some Observations:

  • The pattern of missing values is MNAR (Missing Not At Random) - these were systematically omitted but the aggregate value representing the missing detail is provided
  • Population data for this time frame is deterministic: Population levels rise linearly each year; The duty cycle of a human body is finite and constraints and limits are well known.

Looking at the data we can see that each of the three subsets of years have very similar patterns. More variation in younger ages and variation flattens out for ages greater than 60

All available population values ages 0-99 for years 1940 through 2010

Then if we focus on the years 1980 through 1989 we can fit a nice curve for ages 0 through 100 with a Multiple-R-Squared of .979.

Average of age per each year and fitted curve of 4th order

Then if we narrow the focus to ages 60 to 100 and even narrower to ages 85 to 100 the Multiple-R-Squared increases to .9996.

enter image description here

enter image description here

Now if we flip our focus and look at the increasing levels of population we can observe that these relationships are linear. Population rises at a steady rate year over year.

Total population 1940 through 2010:

enter image description here

Ages 85 through 100 for years 1980 through 1999. Each age is linear. Each age has a slightly lower rate of increase (lesser slope).

enter image description here

My question

This is where I could use some guidance to move forward:

When imputing discrete missing population values by age and year, how do I combine the fitted curve that models changes in population when age increases with the linear regression that models changes in population year over year?

Does one or more documented methods naturally apply to the problem as I have described it? For example: KNN, PCA, BPCA, Mean, MICE, other?

If there is recommended method can you point me to available R or Python packages and documentation that describes the mechanics of applying a given approach?


1 Answer 1


I think you need to be wary of using curves to extrapolate beyond the age thresholds - specifically I think you should consider:

  1. mortality increases with age; I would imagine it increases at an increasing rate with age (especially at higher ages). Would you be able to capture this effectively?
  2. there is obviously some sort of overlap between populations in subsequent years (e.g. someone who is included in 1985 data could also be in 1986 data). What effect would this have on the data imputation?

I would suggest a different approach. Actuaries have traditionally produced "life tables" which capture mortality in population cohorts. If you can find a set of tables which is applicable to the period in question, you could use these to calculate population numbers.

  • $\begingroup$ do you have any thoughts on using the linear models for year over year growth by age? I have a linear equation for age 85 for the actual data from 1980-1999. If I provide 1950 as the independent variable then it will calculate a population value for age 85 for year 1950. This assumes that the linear relationship holds true over the entire period. $\endgroup$
    – Threadid
    Commented Feb 22, 2018 at 17:27
  • $\begingroup$ That might be a good way to tackle the problem; a down-side would be the number of models you have to build (I'm assuming you plan on building a separate model for age 86, 87, and so on?). Maybe also consider non-linear curves? The life table approach I suggested above does something similar, although it uses year-on-year survival probabilities. So my thinking was to source an appropriate life table, and apply the survival probabilities to extrapolate beyond the truncation. $\endgroup$
    – bradS
    Commented Feb 23, 2018 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.