Featured Research

from universities, journals, and other organizations

Researchers share useful lessons learned in evaluating emerging technologies

Date:
June 27, 2011
Source:
National Institute of Standards and Technology (NIST)
Summary:
A team from the U.S. National Institute of Standards and Technology shares critical "lessons learned" that can help businesses and others negotiate the promises and pitfalls encountered when pushing the technology envelope to enable new capabilities.

A NIST-designed framework for evaluating emerging technologies helped to focus and further efforts to develop two-way speech-translation systems to aid US soldiers and Marines serving in Iraq and Afghanistan. In this trial, a Marine and a speaker of Pashto, a native Afghani tongue, conversed using a translation system based on smart phone technology. The technology-development effort was sponsored by the Defense Advanced Research Projects Agency.
Credit: NIST

Most industry executives, military planners, research managers or venture capitalists charged with assessing the potential of an R&D project probably are familiar with the wry twist on Arthur C. Clarke's third law: "Any sufficiently advanced technology is indistinguishable from a rigged demo."

After serving for five years as independent evaluators of emerging military technologies nurtured by the Defense Advanced Research Projects Agency (DARPA), a team from the National Institute of Standards and Technology (NIST) shares critical "lessons learned" that can help businesses and others negotiate the promises and pitfalls encountered when pushing the technology envelope to enable new capabilities.

Writing in the International Journal of Intelligent Control and Systems, the NIST researchers also describe the evaluative framework they devised for judging the performance of a system and its components as well as the utility of the technology for the intended user. Called SCORE (System, Component, and Operationally Relevant Evaluations), the framework is a unified set of criteria and software tools for evaluating emerging technologies from different perspectives and levels of detail and at various stages of development.

SCORE was developed for evaluating so-called intelligent systems -- a fast growing category of technologies ranging from robots and unmanned vehicles to sensor networks, natural language processing devices and "smart" appliances. By definition, explains Craig Schlenoff, acting head of NIST's Systems Integration Group, "Intelligent systems can respond to conditions in an uncertain environment -- be it a battlefield, a factory floor, or an urban highway system -- in ways that help the technology accomplish its intended purpose."

Schlenoff and his colleagues used their SCORE approach to evaluate technologies as they progressed under two DARPA programs: ASSIST and TRANSTAC. In ASSIST, DARPA is funding efforts to instrument soldiers with wearable sensors -- video cameras, microphones, global positioning devices and more -- to continuously record activities while they are on a mission. TRANSTAC is driving the development of two-way speech-translation systems that enable speakers of different languages to communicate with each other in real-world situations, without an interpreter. By providing constructive feedback on system capabilities, the SCORE evaluative framework helps to drive innovation and performance improvements.

Several lessons learned recounted by the NIST team are aimed at maximizing the contributions of test subjects and the developers of technologies without biasing test results. "There is often a balancing act," they write, "between creating the evaluation environment in a way that shows the system in the best possible light vs. having an environment that is as realistic as possible."

They also discuss unavoidable trade-offs due to costs, logistics, or other factors. While evaluators and technology developers should never lose sight of their ultimate objective, the NIST researchers also advise the need for flexibility over time. As more is learned about the system and about user requirements, features may change and project goals may be modified, necessitating adjustments to the evaluation approach.

"The main lesson," Schlenoff explains, "is that the extra effort devoted to evaluation planning can have a huge effect on how successful the evaluation will be. Bad decisions made during the design can be difficult and costly to fix later on."

To learn more about SCORE, go to: http://www.nist.gov/el/isd/ks/score.cfm.


Story Source:

The above story is based on materials provided by National Institute of Standards and Technology (NIST). Note: Materials may be edited for content and length.


Journal Reference:

  1. C. Schlenoff, B. Weiss and M. Steves. A detailed discussion of lessons learned in evaluating emerging and advanced military technologies. International Journal of Intelligent Control and Systems, 2011; (forthcoming)

Cite This Page:

National Institute of Standards and Technology (NIST). "Researchers share useful lessons learned in evaluating emerging technologies." ScienceDaily. ScienceDaily, 27 June 2011. <www.sciencedaily.com/releases/2011/06/110627123042.htm>.
National Institute of Standards and Technology (NIST). (2011, June 27). Researchers share useful lessons learned in evaluating emerging technologies. ScienceDaily. Retrieved August 20, 2014 from www.sciencedaily.com/releases/2011/06/110627123042.htm
National Institute of Standards and Technology (NIST). "Researchers share useful lessons learned in evaluating emerging technologies." ScienceDaily. www.sciencedaily.com/releases/2011/06/110627123042.htm (accessed August 20, 2014).

Share This




More Science & Society News

Wednesday, August 20, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Charter Schools Alter Post-Katrina Landscape

Charter Schools Alter Post-Katrina Landscape

AP (Aug. 20, 2014) Nine years after Hurricane Katrina, charter schools are the new reality of public education in New Orleans. The state of Louisiana took over most of the city's public schools after the killer storm in 2005. (Aug. 20) Video provided by AP
Powered by NewsLook.com
Kids' Drawings At Age 4 Linked To Intelligence At Age 14

Kids' Drawings At Age 4 Linked To Intelligence At Age 14

Newsy (Aug. 19, 2014) A study by King's College London says there's a link between how well kids draw at age 4 and how intelligent they are later in life. Video provided by Newsy
Powered by NewsLook.com
Breakingviews: Ebola's Economic Impact Could Eclipse SARS

Breakingviews: Ebola's Economic Impact Could Eclipse SARS

Reuters - Business Video Online (Aug. 18, 2014) The virus ravaging Africa has yet to spread elsewhere. Yet Asia’s SARS crisis in 2003 showed how changes to behaviour can hurt the economy more than the actual disease, says Breakingviews' Una Galani. Video provided by Reuters
Powered by NewsLook.com
USDA Cracks Down On Imports From Foreign Puppy Mills

USDA Cracks Down On Imports From Foreign Puppy Mills

Newsy (Aug. 18, 2014) New USDA measures to regulate dog imports aim to crack down on buying dogs from overseas puppy mills. Video provided by Newsy
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins