I am trying to create a predictive model that will be built on individual data, but user input will only exist at the group level. Reasoning is that I have 5 million rows of data at the individual level, but only 100 companies exist in that data.

For example: I want to calculate the % increase in earnings an investment firm earns year over year.

My Level 1 data would contain fields such as: Analyst_Id, Company_Name, Age, Earnings_2019, Earnings_2020, Level_of_Education, Company_region, Number_of_Trades

I can use this data to calculate characteristics for each Company_Name: Total_Earnings_2019, Average Investor Age, Average number of trades made, and company region.

The results should be interpreted at the Company level - ie, a user can input a company's average age, region, and average level of education and the output will be a predicted earnings trend for a company with those characteristics.

I've been looking into Multilevel modeling / hierarchical lineal models / nested data structures, but these techniques appear to combine individual & group level variables in the final model, whereas I would be building a model that only has group characteristics.

Which modeling techniques would make sense for this scenario?

Edit: Alternatively, are there ways that I could synthetically generate more companies from my underlying individual data, so that I could train a model on >100 data points at the group level?


1 Answer 1


If I understood the assignment you have given here, one nice idea to use, which is used among football/baseball scouts when wanting to profit, is described by the flow of ideas below:

INPUT) Characteristics of a company are given, let's say age and earnings in this and previous year with number of sales and area of activity.

HISTORIC WINDOW SLIDING) Find, among yours historic data, moments in the past when each company was of similar age to the company given as an input and treat those moments as present information. You can see this as "sliding" the historic data "to the right in time". Let's say you inputed my company that is two years old at the moment - you will "move" the information you collected in your historic data about Facebook so that the info about their business from 2006. is seen as present (they were founded in 2004.) and YouTube so that their business from 2007. is seen as present (they were founded in 2005.).

CLUSTERING) Now that you have context to place your input in, use some clustering method to find companies most similar to your input in features other than age: companies that have similar ratio of sales and activity, companies that have similar human resources, etc etc. This is dependent on the task at hand and will take serious consideration of how to score similarity.

OBSERVE) What is left is to how selected companies acted when they were of the same age as the given company. In my example, you would check how Facebook and YouTube trended in 2007. and 2008., respectively, and treat that as a notion of trend for my company.

This method is good for trend, but not great for calculating the amount of growth and such. It is also important to find a way to include information about macroeconomic aspects into your model as well as general progress and public interest. this is when your good model can become great, but this will take time.

If the data you have are time series of info about your companies, make a time series model that also depends on time context and previous trends, by aggregation and sliding-window analysis. You can then take info about your company as input and try to predict the classifier task of upwards trend/downwards trend or even percentage of growth, depending on the data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.