Featured Research

from universities, journals, and other organizations

Federal Government Should Improve Assessments Of Research Efficiency, Report Notes

Date:
January 31, 2008
Source:
National Academies
Summary:
The White House Office of Management and Budget evaluates research at the US Environmental Protection Agency and other federal agencies using the Program Assessment Rating Tool, a set of questions that asks agencies about many aspects of their programs, including whether they can measure and demonstrate annual improvements in efficiency. Based on the answers, OMB rates research programs as effective, ineffective or somewhere in between. An "ineffective" rating can have serious adverse consequences for a program or agency.

The White House Office of Management and Budget evaluates research at the U.S. Environmental Protection Agency and other federal agencies using the Program Assessment Rating Tool (PART), a set of questions that asks agencies about many aspects of their programs, including whether they can measure and demonstrate annual improvements in efficiency. Based on the answers, OMB rates research programs as effective, ineffective, or somewhere in between. An "ineffective" rating can have serious adverse consequences for a program or agency. After experiencing difficulty meeting OMB's requirements to demonstrate the efficiency of its research programs, EPA asked a National Research Council committee for guidance on how to measure efficiency.

The committee's new report recommends four changes in how the federal government assesses the efficiency of research at EPA and other agencies. First, the report emphasizes that assessing efficiency should be considered only one part of evaluating a program's quality, relevance, and effectiveness.

Second, the report introduces a novel distinction between "investment efficiency" and "process efficiency," and recommends that these aspects be evaluated in different ways. Assessments of investment efficiency should examine whether an agency's R&D portfolio, including the budget, is relevant, of high quality, matches the agency's strategic plan, and is adjusted as new knowledge and priorities emerge.

These evaluations require panels of experts, which should comprise both scientists and other stakeholders. In contrast, evaluations of process efficiency should focus on "inputs" (the people, funds, and facilities dedicated to research) and "outputs" (the services, grants, publications, monitoring, and new techniques produced by research), as well as their timelines. Of nine measures currently used by R&D agencies, common examples are the number of grants awarded or publications produced annually, which could be assessed against appropriate benchmarks.

Third, the committee clarified the roles of outputs and outcomes. EPA's difficulties in answering PART's questions about efficiency have grown out of OMB's insistence that the agency find ways to measure the efficiency of its research based on outcomes, rather than outputs. Measuring research efficiency based on what the committee describes as "ultimate outcomes" -- for example, whether a program eventually results in cleaner air or fewer deaths -- is neither achievable nor valid, because such outcomes occur far in the future and are highly dependent upon actions taken by many other people who may or may not use the research findings. The committee's review of practices across government R&D agencies revealed that no agency has found a way to demonstrate efficiency based on ultimate outcomes.

The report does endorse evaluating efficiency based on "intermediate outcomes" -- for example, assessing whether a program has improved the body of knowledge for decision-making, or disseminated newly developed tools and models. These assessments would be conducted most effectively using expert review panels and not just formulas.

Finally, the committee recommended that the efficiency of EPA's research programs be evaluated according to the same standards used at other agencies. OMB has rejected some methods for measuring research efficiency when proposed by EPA, but accepted them when proposed by other agencies. OMB should train and oversee its budget examiners to make sure they implement the PART questionnaire consistently and equitably across agencies.

The study was sponsored by the U.S. Environmental Protection Agency. The National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council make up the National Academies. They are private, nonprofit institutions that provide science, technology, and health policy advice under a congressional charter.


Story Source:

The above story is based on materials provided by National Academies. Note: Materials may be edited for content and length.


Cite This Page:

National Academies. "Federal Government Should Improve Assessments Of Research Efficiency, Report Notes." ScienceDaily. ScienceDaily, 31 January 2008. <www.sciencedaily.com/releases/2008/01/080131164703.htm>.
National Academies. (2008, January 31). Federal Government Should Improve Assessments Of Research Efficiency, Report Notes. ScienceDaily. Retrieved October 21, 2014 from www.sciencedaily.com/releases/2008/01/080131164703.htm
National Academies. "Federal Government Should Improve Assessments Of Research Efficiency, Report Notes." ScienceDaily. www.sciencedaily.com/releases/2008/01/080131164703.htm (accessed October 21, 2014).

Share This



More Earth & Climate News

Tuesday, October 21, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Traditional Farming Methods Gaining Ground in Mali

Traditional Farming Methods Gaining Ground in Mali

AFP (Oct. 20, 2014) He is leading a one man agricultural revolution in Mali - Oumar Diatabe uses traditional farming methods to get the most out of his land and is teaching others across the country how to do the same. Duration: 01:44 Video provided by AFP
Powered by NewsLook.com
Hey, Doc! Sewage, Beer and Food Scraps Can Power Chevrolet’s Bi-Fuel Impala

Hey, Doc! Sewage, Beer and Food Scraps Can Power Chevrolet’s Bi-Fuel Impala

3BL Media (Oct. 20, 2014) Hey, Doc! Sewage, Beer and Food Scraps Can Power Chevrolet’s Bi-fuel Impala Video provided by 3BL
Powered by NewsLook.com
White Rhino's Death In Kenya Means Just 6 Are Left

White Rhino's Death In Kenya Means Just 6 Are Left

Newsy (Oct. 20, 2014) Suni, a rare northern white rhino at Ol Pejeta Conservancy, died Friday. This, as many media have pointed out, leaves people fearing extinction. Video provided by Newsy
Powered by NewsLook.com
New Organic Fertilizer Helps Reforestation of Monarch Butterflies’ Winter Retreat

New Organic Fertilizer Helps Reforestation of Monarch Butterflies’ Winter Retreat

Reuters - Innovations Video Online (Oct. 20, 2014) Using an organic fertiliser, a conservationist from the National Autonomous University of Mexico (UNAM), leads an award-winning project to reforest the sanctuary of monarch butterflies. Sharon Reich reports. Video provided by Reuters
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:

Strange & Offbeat Stories


Plants & Animals

Earth & Climate

Fossils & Ruins

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins