I'm trying to predict the population for states and the country in 2050. My current dataset has values for each state from 1951,1961...2011 in the same table. Here is a sample view:

Row States/Union Territories         1951   1961    1971    1981    1991    2001    2011
0   Andaman and Nicobar Islands   31    64       115     189     281     356    381
1   Andhra Pradesh               31115  35983   43503   53551   66508   76210   84581   
2   Arunachal Pradesh             307   337     468     632     865     1098    1384    

So i ran a simple regression for values in 2011 and the model works very well. My question here is, how do i run it for an entirely new column(2050), which has no data to compare with in order to test for accuracy or any other metrics? One thing i intuitively tried was to add a new column as 2050 and put all the values as 0, but then even the predicted values were 0 so that's of no help. I'm new to this so thanks for any help!

Associated code:

data = pd.read_csv("final_doc.csv", encoding = "latin-1")
data['2050'] = 0
X = data[['1951', '1961', '1971', '1981', '1991', '2001']].values
Y = data[['2011']].values
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2)
regressor = LinearRegression()
regressor.fit(X_train, Y_train)
y_pred = regressor.predict(X_test)
fin = pd.DataFrame(y_pred, columns = ['2011'])

3 Answers 3


The form of your model is incorrectly specified. You want to model your data so that your independent variables are year (a numeric variable) and state/union (a categorical variable). Your dependent variable will be the population size.

In order to develop such a model you will need to restructure your data so that it is in a long format (see below)

States/Union Territories      Year                Pop 
Andaman and Nicobar Islands   1951                 31
Andaman and Nicobar Islands   1961                 64
           ...                 ...                ...
Andhra Pradesh                1951              31115   
Andhra Pradesh                1961              35983
           ...                 ...                ...   
Arunachal Pradesh             1951                307
           ...                 ...                ...

Once the data is transformed into a proper format, you can then fit your model. I should mention you will need to recode your categorical variable - you can use a method called one hot-encoding to achieve this.

Once your model has been fit, you will be able to predict the expected population size for a given year and Union/State.

The following post Multiple Linear Regression Model in 7 Steps with Python provides a great guide to developing a model.

Hope this helps!

  • $\begingroup$ After restructuring the table in the way you specified, i will have 231 rows.. That means a 8 bit vector as my one hot encoding, am i right? $\endgroup$ Apr 22, 2020 at 7:23
  • $\begingroup$ It would also be worth to look into one-hot encoding of the "year" variable, since treating year as numeric imposes a linear trend over time. One-hot encoding could lead to a better fit. $\endgroup$
    – Peter
    Apr 22, 2020 at 8:01
  • $\begingroup$ @Peter I can only use linear regression as specified by my professor, whether it's a good fit or not $\endgroup$ Apr 22, 2020 at 8:17
  • $\begingroup$ You can use dummy/one hot encoding (for years) in linear regression. This is a common practice. $\endgroup$
    – Peter
    Apr 22, 2020 at 8:49
  • $\begingroup$ @Peter but nwaldo stated that i should use one hot encoding for the categorical data instead, i.e the states/UT's. This leaves me confused due to conflicting opinions. Could you care to explain? $\endgroup$ Apr 22, 2020 at 9:03

You essentially look at population growth and consequently you could model growth rates directly. Here is a example in R.

First, get some data (China in this case):

# Pop China rounded (mio.)
# https://en.wikipedia.org/wiki/Demographics_of_China
df = data.frame(c(547,609,667,715,818,916,981,1045),c(1950,1955,1960,1965,1970,1975,1980,1985))

This looks like:

   pop year
1  547 1950
2  609 1955
3  667 1960
4  715 1965
5  818 1970
6  916 1975
7  981 1980
8 1045 1985

Next I run a "normal" linear regression (level-level):

# Normal linear regression "level-level"
reg1 = lm(pop~year,data=df)

# Predict / plot result
pred1 = predict(reg1, newdata=df)
plot(df$year, pred1, type="b")
lines(df$year, df$pop, type = "o", col = "blue")

Results are not bad: Adjusted R-squared: 0.9891. Also the plot (actual vs. predicted) looks really good.

enter image description here

However, we can model growth rates directly using a "log-level" specification. In this case, the dependent variable $y$ simply is "transformed" to $log(y)$, so that the regression model looks like:

$$ log(y) = \beta_0 + \beta_1 x + u .$$

Note that the interpretation of $\beta_1$ has changed now. If you increase $x$ (the year) by one unit, $y$ changes (c.p. on average) by $100(\exp^{\beta_1}-1)$ percent. This is the average population growth rate in the period 1950-1985. Here in this example, the average growth rate was about 1.9 percent per year.

# Linear regression log-level
reg2 = lm(log(pop)~year,data=df)

# The average growth rate

# Predict / plot result
pred2 = exp(predict(reg2, newdata=df))
plot(df$year, pred2, type="b")
lines(df$year, df$pop, type = "o", col = "blue")

Note that we see a slightly better model fit here, since log-level transformations usually can smooth out a little bit of "wonkiness" in the data. We can also look at the plot.

enter image description here

Next we can move to "out of sample" prediction until 2015.

# "Out of sample" prediction
preddf = data.frame(c(1950,1955,1960,1965,1970,1975,1980,1985,1990,1995,2000,2005,2010,2015))
pred3 = exp(predict(reg2, newdata=preddf))

# Actual figures
actual = c(547,609,667,715,818,916,981,1045,1135,1204,1263,1304,1338,1375)
result = data.frame(cbind(preddf,round(pred3),actual))

# Plot results
plot(result$year, result$pred, type="b")
lines(result$year, result$act, type = "o", col = "blue")

The results look like:

   year pred  act
1  1950  551  547
2  1955  606  609
3  1960  666  667
4  1965  733  715
5  1970  806  818
6  1975  887  916
7  1980  976  981
8  1985 1073 1045
9  1990 1180 1135
10 1995 1298 1204
11 2000 1428 1263
12 2005 1571 1304
13 2010 1728 1338
14 2015 1900 1375

enter image description here

Wow... Bonkers! What happend? Well, aparently China's population growth did change. Growth decreased and we are not able to capture this trend in the data. What could be done? Maybe use (non-linear) generalised additive models, e.g. with regression splines. However, this will likely give poor results. Alternatively, I could estimate future growth only based on the last three or four observations, assuming that they are most representative for future growth. Note the differend trend in population growth between 1975 and 1985 (looks "flatter"). We could use this more recent patterns to get a better estimate.

Find an application of this technique (log-level linear regression) to Covid-19 on this Github.

Splendid! Someone just did your homework ;-)

  • $\begingroup$ Thanks so much for the effort you have put into this answer, however im still at a loss on how to actually implement this in Pandas and sklearn, which was my original question. I understand that you've explained the logic behind approaching it which i really appreciate, but my question is still unanswered. I HAVE to implement is using linear regression since those are the course specifications, so changing the model is of no use. I was really looking for some exact implementation details for prediction of a future year . Thank you once again for a great answer, $\endgroup$ Apr 22, 2020 at 17:05
  • $\begingroup$ The example above is linear regression $\endgroup$
    – Peter
    Apr 22, 2020 at 17:21
  • $\begingroup$ But it's in R. I said i needed implementation help using pandas and sklearn specifically, thanks! $\endgroup$ Apr 22, 2020 at 17:30
  • $\begingroup$ I can‘t do all of your homework... $\endgroup$
    – Peter
    Apr 22, 2020 at 17:56
  • $\begingroup$ I'm not asking you to do any of it. I need explicit help with a particular issue which i have explained in my question very clearly. $\endgroup$ Apr 23, 2020 at 5:21

This is exactly what I was looking for. In short, the indices are set as the states and then transposed. After which linear regression can be simply applied eliminating the need for one-hot encoding. Thank you everyone for your help.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.