Featured Research

from universities, journals, and other organizations

Researchers share useful lessons learned in evaluating emerging technologies

Date:
June 27, 2011
Source:
National Institute of Standards and Technology (NIST)
Summary:
A team from the U.S. National Institute of Standards and Technology shares critical "lessons learned" that can help businesses and others negotiate the promises and pitfalls encountered when pushing the technology envelope to enable new capabilities.

A NIST-designed framework for evaluating emerging technologies helped to focus and further efforts to develop two-way speech-translation systems to aid US soldiers and Marines serving in Iraq and Afghanistan. In this trial, a Marine and a speaker of Pashto, a native Afghani tongue, conversed using a translation system based on smart phone technology. The technology-development effort was sponsored by the Defense Advanced Research Projects Agency.
Credit: NIST

Most industry executives, military planners, research managers or venture capitalists charged with assessing the potential of an R&D project probably are familiar with the wry twist on Arthur C. Clarke's third law: "Any sufficiently advanced technology is indistinguishable from a rigged demo."

After serving for five years as independent evaluators of emerging military technologies nurtured by the Defense Advanced Research Projects Agency (DARPA), a team from the National Institute of Standards and Technology (NIST) shares critical "lessons learned" that can help businesses and others negotiate the promises and pitfalls encountered when pushing the technology envelope to enable new capabilities.

Writing in the International Journal of Intelligent Control and Systems, the NIST researchers also describe the evaluative framework they devised for judging the performance of a system and its components as well as the utility of the technology for the intended user. Called SCORE (System, Component, and Operationally Relevant Evaluations), the framework is a unified set of criteria and software tools for evaluating emerging technologies from different perspectives and levels of detail and at various stages of development.

SCORE was developed for evaluating so-called intelligent systems -- a fast growing category of technologies ranging from robots and unmanned vehicles to sensor networks, natural language processing devices and "smart" appliances. By definition, explains Craig Schlenoff, acting head of NIST's Systems Integration Group, "Intelligent systems can respond to conditions in an uncertain environment -- be it a battlefield, a factory floor, or an urban highway system -- in ways that help the technology accomplish its intended purpose."

Schlenoff and his colleagues used their SCORE approach to evaluate technologies as they progressed under two DARPA programs: ASSIST and TRANSTAC. In ASSIST, DARPA is funding efforts to instrument soldiers with wearable sensors -- video cameras, microphones, global positioning devices and more -- to continuously record activities while they are on a mission. TRANSTAC is driving the development of two-way speech-translation systems that enable speakers of different languages to communicate with each other in real-world situations, without an interpreter. By providing constructive feedback on system capabilities, the SCORE evaluative framework helps to drive innovation and performance improvements.

Several lessons learned recounted by the NIST team are aimed at maximizing the contributions of test subjects and the developers of technologies without biasing test results. "There is often a balancing act," they write, "between creating the evaluation environment in a way that shows the system in the best possible light vs. having an environment that is as realistic as possible."

They also discuss unavoidable trade-offs due to costs, logistics, or other factors. While evaluators and technology developers should never lose sight of their ultimate objective, the NIST researchers also advise the need for flexibility over time. As more is learned about the system and about user requirements, features may change and project goals may be modified, necessitating adjustments to the evaluation approach.

"The main lesson," Schlenoff explains, "is that the extra effort devoted to evaluation planning can have a huge effect on how successful the evaluation will be. Bad decisions made during the design can be difficult and costly to fix later on."

To learn more about SCORE, go to: http://www.nist.gov/el/isd/ks/score.cfm.


Story Source:

The above story is based on materials provided by National Institute of Standards and Technology (NIST). Note: Materials may be edited for content and length.


Journal Reference:

  1. C. Schlenoff, B. Weiss and M. Steves. A detailed discussion of lessons learned in evaluating emerging and advanced military technologies. International Journal of Intelligent Control and Systems, 2011; (forthcoming)

Cite This Page:

National Institute of Standards and Technology (NIST). "Researchers share useful lessons learned in evaluating emerging technologies." ScienceDaily. ScienceDaily, 27 June 2011. <www.sciencedaily.com/releases/2011/06/110627123042.htm>.
National Institute of Standards and Technology (NIST). (2011, June 27). Researchers share useful lessons learned in evaluating emerging technologies. ScienceDaily. Retrieved September 18, 2014 from www.sciencedaily.com/releases/2011/06/110627123042.htm
National Institute of Standards and Technology (NIST). "Researchers share useful lessons learned in evaluating emerging technologies." ScienceDaily. www.sciencedaily.com/releases/2011/06/110627123042.htm (accessed September 18, 2014).

Share This



More Science & Society News

Thursday, September 18, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

The Cost of Ebola

The Cost of Ebola

Reuters - Business Video Online (Sep. 18, 2014) As Sierra Leone prepares for a three-day "lockdown" in its latest bid to stem the spread of Ebola, Ciara Lee looks at the financial implications of fighting the largest ever outbreak of the disease. Video provided by Reuters
Powered by NewsLook.com
Stocks Hit All-Time High as Fed Holds Steady

Stocks Hit All-Time High as Fed Holds Steady

AP (Sep. 17, 2014) The Federal Reserve signaled Wednesday that it plans to keep a key interest rate at a record low because a broad range of U.S. economic measures remain subpar. Stocks hit an all-time high on the news. (Sept. 17) Video provided by AP
Powered by NewsLook.com
Some Tobacco Farmers Thrive Amid Challenges

Some Tobacco Farmers Thrive Amid Challenges

AP (Sep. 16, 2014) The South's tobacco country is surviving, and even thriving in some cases, as demand overseas keeps growers in the fields of one of America's oldest cash crops. (Sept. 16) Video provided by AP
Powered by NewsLook.com
Space Race Pits Bezos Vs Musk

Space Race Pits Bezos Vs Musk

Reuters - Business Video Online (Sep. 16, 2014) Amazon CEO Jeff Bezos' startup will team up with Boeing and Lockheed to develop rocket engines as Elon Musk races to have his rockets certified. Fred Katayama reports. Video provided by Reuters
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins