Featured Research

from universities, journals, and other organizations

Overcoming structural uncertainty in computer models

Date:
April 1, 2014
Source:
Society for Industrial and Applied Mathematics
Summary:
A computer model is a representation of the functional relationship between one set of parameters, which forms the model input, and a corresponding set of target parameters, which forms the model output. A true model for a particular problem can rarely be defined with certainty. The most we can do to mitigate error is to quantify the uncertainty in the model. Scientists have now offered a method to incorporate judgments into a model about structural uncertainty that results from building an 'incorrect' model.

Left: Hypothetical model with ten inputs and one output, decomposed to reveal six intermediate parameters. Right: Possible structural error in the subfunctions that result in Y1, Y5, and Y6 are corrected with discrepancy terms δ1, δ2 and δ3.
Credit: Mark Strong and Jeremy E. Oakley

A computer model is a representation of the functional relationship between one set of parameters, which forms the model input, and a corresponding set of target parameters, which forms the model output. A true model for a particular problem can rarely be defined with certainty. The most we can do to mitigate error is to quantify the uncertainty in the model.

In a recent paper published in the SIAM/ASA Journal on Uncertainty Quantification, authors Mark Strong and Jeremy Oakley offer a method to incorporate judgments into a model about structural uncertainty that results from building an "incorrect" model.

"Given that 'all models are wrong,' it is important that we develop methods for quantifying our uncertainty in model structure such that we can know when our model is 'good enough'," author Mark Strong says. "Better models mean better decisions."

When making predictions using computer models, we encounter two sources of uncertainty: uncertainty in model inputs and uncertainty in model structure. Input uncertainty arises when we are not certain about input parameters in model simulations. If we are uncertain about true structural relationships within a model -- that is, the relationship between the set of quantities that form the model input and the set that represents the output -- the model is said to display structural uncertainty. Such uncertainty exists even if the model is run using input values as estimated in a perfect study with infinite sample size.

"Perhaps the hardest problem in assessing uncertainty in a computer model prediction is to quantify uncertainty about the model structure, particularly when models are used to predict in the absence of data," says author Jeremy Oakley. "The methodology in this paper can help model users prioritize where improvements are needed in a model to provide more robust support to decision making."

While methods for managing input uncertainty are well described in the literature, methods for quantifying structural uncertainty are not as well developed. This is especially true in the context of health economic decision making, which is the focus of this paper. Here, models are used to predict future costs and health consequences of options to make decisions for resource allocation.

"In health economics decision analysis, the use of "law-based" computer models is common. Such models are used to support national health resource allocation decisions, and the stakes are therefore high," says Strong. "While it is usual in this setting to consider the uncertainty in model inputs, uncertainty in model structure is almost never formally assessed."

There are several approaches to managing model structural uncertainty. A primary approach is 'model averaging' in which predictions of a number of plausible models are averaged with weights based on each model's likelihood or predictive ability. Another approach is 'model calibration', which assesses a model based on its external discrepancies, that is, output quantities and how they relate to real, observed values. In the context of healthcare decisions, however, neither of these approaches is feasible since typically more than one model is not available for averaging, and observations on model outputs are not available for calibration.

Hence, the authors use a novel approach based on discrepancies within the model or "internal discrepancies" (as opposed to external discrepancies which are the focus of model calibration). Internal discrepancies are analyzed by first decomposing the model into a series of subunits or subfunctions, the outputs of which are intermediate model parameters that are potentially observable in the real world. Next, each sub-function is judged for certainty based on whether its output would equal the true value of the parameter from real-world observations. If a potential structural error is anticipated, a discrepancy term is introduced. Subsequently, beliefs about the size and direction of errors are expressed. Since judgments for internal discrepancies are expected to be crude at best, the expression of uncertainty should be generous, that is, allowed to cover a wide distribution of possible values. Finally, the authors determine the sensitivity of the model output to internal discrepancies. This gives an indication of the relative importance of structural uncertainty within each model subunit.

"Traditional statistical approaches to handling uncertainty in computer models have tended to treat the models as 'black boxes'. Our framework is based on 'opening' the black box and investigating the model's internal workings," says Oakley. "Developing and implementing this framework, particularly in more complex models, will need closer collaboration between statisticians and mathematical modelers."


Story Source:

The above story is based on materials provided by Society for Industrial and Applied Mathematics. Note: Materials may be edited for content and length.


Journal Reference:

  1. Mark Strong, Jeremy E. Oakley. When Is a Model Good Enough? Deriving the Expected Value of Model Improvement via Specifying Internal Model Discrepancies. SIAM/ASA Journal on Uncertainty Quantification, 2014; 2 (1): 106 DOI: 10.1137/120889563

Cite This Page:

Society for Industrial and Applied Mathematics. "Overcoming structural uncertainty in computer models." ScienceDaily. ScienceDaily, 1 April 2014. <www.sciencedaily.com/releases/2014/04/140401112121.htm>.
Society for Industrial and Applied Mathematics. (2014, April 1). Overcoming structural uncertainty in computer models. ScienceDaily. Retrieved September 14, 2014 from www.sciencedaily.com/releases/2014/04/140401112121.htm
Society for Industrial and Applied Mathematics. "Overcoming structural uncertainty in computer models." ScienceDaily. www.sciencedaily.com/releases/2014/04/140401112121.htm (accessed September 14, 2014).

Share This



More Computers & Math News

Sunday, September 14, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Frustration As Drone Industry Outpaces Regulation In U.S.

Frustration As Drone Industry Outpaces Regulation In U.S.

Newsy (Sep. 14, 2014) U.S. firms worry they’re falling behind in the marketplace as the FAA considers how to regulate commercial drones. Video provided by Newsy
Powered by NewsLook.com
iPhone 6 Sales Mark Yet Another Year Of Records, Glitches

iPhone 6 Sales Mark Yet Another Year Of Records, Glitches

Newsy (Sep. 13, 2014) Customers looking to preorder the iPhone 6 on Friday experienced a few hiccups thanks to record demand for the device overnight. Video provided by Newsy
Powered by NewsLook.com
Is Photo-Sharing App Tiiny Really A Snapchat Competitor?

Is Photo-Sharing App Tiiny Really A Snapchat Competitor?

Newsy (Sep. 13, 2014) Tiiny, a photo-sharing app, is being called a Snapchat competitor. But after testing it ourselves, we'd have to disagree. Video provided by Newsy
Powered by NewsLook.com
Ebola Batters Sierra Leone Economy Too

Ebola Batters Sierra Leone Economy Too

Reuters - Business Video Online (Sep. 12, 2014) The World Health Organisation warns that local health workers in West Africa can't keep up with Ebola - and among those countries hardest hit by the outbreak, the economic damage is coming into focus, too. As David Pollard reports, Sierra Leone admits that growth in one of the poorest economies in the region is taking a beating. Video provided by Reuters
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

    Technology News



    Save/Print:
    Share:

    Free Subscriptions


    Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

    Get Social & Mobile


    Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

    Have Feedback?


    Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
    Mobile: iPhone Android Web
    Follow: Facebook Twitter Google+
    Subscribe: RSS Feeds Email Newsletters
    Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins