So I've this model that simulates an ecosystem and outputs its attributes, like its chemistry, temperature etc. There are lots of input parameters to the model.

My job is to write a program to figure out the values of those parameters automatically using machine learning techniques. i.e to make a guess, run the simulation, then check the results against actual historically observed field data. If the results are very close to the field data, then the parameters are probably correct. If they are off, then I make some adjustment of the parameters and run again. Every parameter has a default value, and can be varied only by +\- 30% of its default value.

There around 30 input parameters to the simulation. However, only 8-10 are candidates for estimation. The simulation takes around 5 minutes to run.

Is this a Parameter estimation problem? I know of few algorithms meant for parameter estimation, like MCMC & Simulated annealing. Are they suitable in this case?

I could easily come up with a naive implementation for varying the parameters' values. Could someone guide/suggest me an approach to come up with an efficient solution?

  • $\begingroup$ how long does it take to run a simulation? when you say lots of parameters, do you mean hundreds, thousands? $\endgroup$ – oW_ Jul 17 '17 at 17:35
  • $\begingroup$ Not so many, around 30 input parameters. out of which only 8-10 are candidates for estimation. The simulation takes around 5 minutes to run. $\endgroup$ – Alex Jul 17 '17 at 18:04
  • $\begingroup$ 5 minutes per simulation run to check values is a major constraint. It rules out some of the fast-and-dumb solvers, such as simulated annealing. $\endgroup$ – Neil Slater Jul 17 '17 at 18:27
  • $\begingroup$ Thanks for the reply. We have plans to parallelise the application and run it on computers with really large number of cores. How do I go about using simulated annealing? $\endgroup$ – Alex Jul 17 '17 at 20:20
  • $\begingroup$ I'm still not sure about simulated annealing for your problem, although it might work. I've only used it on large combinatorial problems with very different characteristics, so I don't think I can answer your question well enough. It may help if you can give some characteristics of your problem space - for instance do the attributes vary smoothly with input parameters (if yes this points to hill-climbing or other gradient-based approaches as feasible)? Checking a course grid of e.g. -20%, 0, +20% for 10 params would cost you nearly 60,000 simulations ~ 200 CPU days. $\endgroup$ – Neil Slater Jul 17 '17 at 20:53

Strategies for evaluating models similar to yours (from Ecology) are discussed in this paper "Facilitating Parameter Estimation and Sensitivity Analysis of Agent-Based Models: A Cookbook Using NetLogo and R" by Thiele et al, 2014. It is about, well, Agent-Based Models and R, but the theoretical aspects are generally applicable.

Just by chance I have read this last weekend. Found it easy to read.

For a fast but maybe suboptimal strategy, what about Latin Hypercube Sampling?, see sect 2.25.

Have not tried it myself, though.

  • $\begingroup$ Thanks a lot for the link to paper! It has a lot of good information for a problem like mine. I am stuck at writing the optimization function, my goal is to reduce the RMSE of model vs observed output. and all I have is the model and observed outputs, do you think it is sufficient? As far as I know, one needs a training set. $\endgroup$ – Alex Jul 21 '17 at 16:02
  • $\begingroup$ Can't you split the observed outputs into a training set and a test set, say 70%/30%? On the training set you perform 10-fold cross validation. When you are done you apply the tuned model on the test set. By the way, you can upvote answers you like. $\endgroup$ – knb Jul 21 '17 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.