How to solve (or even effectively think about) a complex real-life multivariate problem

In my work, we get estimates.

An estimate may include up to 12 different categories of costs (Development, Legal Travel, etc.) to produce any number of assets/deliverables from dozens of different deliverable categories (TV Spot, Radio, Podcast, Talent Session, etc.). Each estimate will come with a quantity (simple integer count) of each type of deliverable received for the sum of all the costs in the estimate. I am attempting to derive a decent/ballpark (not perfect) "cost per" model or function for deliverables. "I want to produce 5 Videos and 4 Radio Spots and 6 Digital Banners....about how much can I expect this to cost?"

The problem is that estimates almost always contain multiple types of deliverables, and you do not have the pleasure of seeing which costs within each estimate apply to which deliverable. Some costs might seem to match a deliverable. For example, you might have an estimate with "Video" deliverables and a "Video Production" cost, but you can't assume any cost applies to a specific deliverable. You simply have a TOTAL for the estimate, and a count of each deliverable asset that you get. Of course, the cost to produce a Video is almost always going to be higher than the cost to produce a photograph, and so I'm hoping that over many samples, I can take advantage of those generalities. But there are many variable.

We do not have enough samples (under 10k) to build a neural network or for this as the dimensions and feature set are simply far too wide. We can't use a regression effectively for the same reason - too many dimensions for the sample size.

So I'm trying to think about this algebraically. Is there a system of equations and/or matrix method I can use, over thousands of estimates like these, to derive a "cost per" for a given deliverable type?

Examples:

Project 1
├── Estimate #1903  $16,443 │ ├── Cost Breakdown │ │ ├── Animation & VFX:$3,675
│   │   ├── Audio & Music:  $3,235 │ │ ├── Development:$8,498
│   │   └── Talent:  $1,036 │ └── Deliverable Breakdown │ ├── Animation/Motion Graphics - 2D Animated Video │ │ └── Social --- 2 Orig. │ ├── Audio/Record & Mix - VO Recording │ │ └── Audio --- 2 Orig. │ ├── Design & Post Production - Retouched Image(s) │ │ └── Social --- 1 Orig. │ └── Online Advertising - Static Banner │ └── Display --- 22 Orig. └── Estimate #1907$16,443
├── Cost Breakdown
│   ├── Animation & VFX:  $3,675 │ ├── Audio & Music:$3,235
│   ├── Development:  $8,498 │ └── Talent:$1,036
└── Deliverable Breakdown
├── Animation/Motion Graphics - 2D Animated Video
│   └── Social --- 1 Orig.
├── Audio/Record & Mix - VO Recording
│   └── Audio --- 2 Orig.
├── Design & Post Production - Retouched Image(s)
│   └── Social --- 1 Orig.
└── Online Advertising - Static Banner
└── Display --- 22 Orig.
Project 2
├── Estimate #1013  $915,855 │ ├── Cost Breakdown │ │ ├── Audio & Music:$43,060
│   │   ├── Editorial & Finishing:  $164,725 │ │ ├── Miscellaneous:$24,075
│   │   ├── Services:  $9,280 │ │ ├── Talent:$59,457
│   │   └── Video Production:  $615,258 │ └── Deliverable Breakdown │ └── Live Action Production - Video │ ├── Native --- 3 Orig. │ ├── Social --- 4 Orig. │ └── TV --- 4 Orig. ├── Estimate #1063$30,950
│   ├── Cost Breakdown
│   │   ├── Audio & Music:  $3,100 │ │ ├── Editorial & Finishing:$27,350
│   │   └── Miscellaneous:  $500 │ └── Deliverable Breakdown │ └── Live Action Production - Video │ ├── Social --- 4 Orig. │ └── TV --- 4 Orig. ├── Estimate #1064$1,812
│   ├── Cost Breakdown
│   │   └── Audio & Music:  $1,812 │ └── Deliverable Breakdown │ ├── Editorial & Finishing - Edited Animatic │ │ └── Testing/Focus Group --- 3 Orig. │ └── Live Action Production - Video │ ├── Native --- 02 Orig. │ ├── Social --- 4 Orig. │ └── TV --- 4 Orig. └── Estimate #1065$27,675
├── Cost Breakdown
│   ├── Audio & Music:  $4,000 │ ├── Editorial & Finishing:$23,175
│   └── Miscellaneous:  \$500
└── Deliverable Breakdown
└── Editorial & Finishing - Edited Animatic
└── Testing/Focus Group --- 3 Orig.

I would suggest using a genetic algorithm of some kind. The idea is to assign hypothetical costs to each item, then check how well the hypothesis matches the data you have. An individual represents an "hypothesis", i.e. assignment of costs: starting from random hypotheses, the genetic algorithm might be able to converge to a solution.

I gave the following more detailed answer to a similar problem a while back:

Clearly this problem doesn't always have a unique solution, but if you are interested in finding one possible solution you could try a simple genetic algorithm simulation:

• Each individual gene represents an item from the list of all possible items.
• Each gene/item is assigned a price randomly at first (gene expression)
• When a mutation is applied to a gene/item, its price is slightly modified randomly.
• A crossover causes a "child gene" to take as value the mean of its two "parents genes".

This setting means that every individual in a population consists of all the items being assigned a particular price. At each generation each individual/assignment is evaluated by applying the prices assignment to the actual data and then measuring the error compared to the actual prices. Finally the top N individuals/assignments which perform the best are selected as parents for the next generation. Eventually the population should converge to realistic prices assignments.

I think this is a perfect case for a genetic algorithm because the evaluation of a potential price assignment is a very simple calculation, so there is no major efficiency issue repeating the process over many generations (as opposed to many problems where evaluation is prohibitively expensive).

• Hadn't thought of a genetic algorithm - I really appreciate this suggestion and will do more reading. When you said, "..and then measuring the error compared to the actual prices," how exactly would I do that in this case? If I don't know the label - the actual cost of an individual "Live Action Production", for example - what do I test my hypothetical cost assignment for a "Live Action Production" to for accuracy? – Zachary Girson Jun 9 at 17:49
• Also, I wonder how something similar to a GAN might be used to tackle an issue like this... could I generate a hypothetical costs for each deliverable and then test the sum of each individual hypothetical deliverable's cost in a given estimate against the known sum of the costs in that estimate? Or something to that effect.. towardsdatascience.com/… – Zachary Girson Jun 9 at 18:03
• @ZacharyGirson an hypothesis contains the individual price for every item. Your data contains examples of the total price estimate for several items together, right? so if an hypothesis is "correct" the sum of a particular group of items should give approximately the same value as the total price from your data. From this idea you can calculate for instance a mean absolute error across all the examples, and select the hypotheses which have the lowest error. – Erwan Jun 10 at 12:53
• I'm not familiar with GAN, you might be right I don't know. – Erwan Jun 10 at 12:57
• The idea would be to make a model that attempts to predict the individual costs based on the different deliverables and then pit the accuracy against the accuracy of a model that is trying to use those individual predicted costs from the other model to predict the deliverables... and make them compete, basically. – Zachary Girson Jun 15 at 15:03