Featured Research

from universities, journals, and other organizations

Researchers share useful lessons learned in evaluating emerging technologies

Date:
June 27, 2011
Source:
National Institute of Standards and Technology (NIST)
Summary:
A team from the U.S. National Institute of Standards and Technology shares critical "lessons learned" that can help businesses and others negotiate the promises and pitfalls encountered when pushing the technology envelope to enable new capabilities.

A NIST-designed framework for evaluating emerging technologies helped to focus and further efforts to develop two-way speech-translation systems to aid US soldiers and Marines serving in Iraq and Afghanistan. In this trial, a Marine and a speaker of Pashto, a native Afghani tongue, conversed using a translation system based on smart phone technology. The technology-development effort was sponsored by the Defense Advanced Research Projects Agency.
Credit: NIST

Most industry executives, military planners, research managers or venture capitalists charged with assessing the potential of an R&D project probably are familiar with the wry twist on Arthur C. Clarke's third law: "Any sufficiently advanced technology is indistinguishable from a rigged demo."

After serving for five years as independent evaluators of emerging military technologies nurtured by the Defense Advanced Research Projects Agency (DARPA), a team from the National Institute of Standards and Technology (NIST) shares critical "lessons learned" that can help businesses and others negotiate the promises and pitfalls encountered when pushing the technology envelope to enable new capabilities.

Writing in the International Journal of Intelligent Control and Systems, the NIST researchers also describe the evaluative framework they devised for judging the performance of a system and its components as well as the utility of the technology for the intended user. Called SCORE (System, Component, and Operationally Relevant Evaluations), the framework is a unified set of criteria and software tools for evaluating emerging technologies from different perspectives and levels of detail and at various stages of development.

SCORE was developed for evaluating so-called intelligent systems -- a fast growing category of technologies ranging from robots and unmanned vehicles to sensor networks, natural language processing devices and "smart" appliances. By definition, explains Craig Schlenoff, acting head of NIST's Systems Integration Group, "Intelligent systems can respond to conditions in an uncertain environment -- be it a battlefield, a factory floor, or an urban highway system -- in ways that help the technology accomplish its intended purpose."

Schlenoff and his colleagues used their SCORE approach to evaluate technologies as they progressed under two DARPA programs: ASSIST and TRANSTAC. In ASSIST, DARPA is funding efforts to instrument soldiers with wearable sensors -- video cameras, microphones, global positioning devices and more -- to continuously record activities while they are on a mission. TRANSTAC is driving the development of two-way speech-translation systems that enable speakers of different languages to communicate with each other in real-world situations, without an interpreter. By providing constructive feedback on system capabilities, the SCORE evaluative framework helps to drive innovation and performance improvements.

Several lessons learned recounted by the NIST team are aimed at maximizing the contributions of test subjects and the developers of technologies without biasing test results. "There is often a balancing act," they write, "between creating the evaluation environment in a way that shows the system in the best possible light vs. having an environment that is as realistic as possible."

They also discuss unavoidable trade-offs due to costs, logistics, or other factors. While evaluators and technology developers should never lose sight of their ultimate objective, the NIST researchers also advise the need for flexibility over time. As more is learned about the system and about user requirements, features may change and project goals may be modified, necessitating adjustments to the evaluation approach.

"The main lesson," Schlenoff explains, "is that the extra effort devoted to evaluation planning can have a huge effect on how successful the evaluation will be. Bad decisions made during the design can be difficult and costly to fix later on."

To learn more about SCORE, go to: http://www.nist.gov/el/isd/ks/score.cfm.


Story Source:

The above story is based on materials provided by National Institute of Standards and Technology (NIST). Note: Materials may be edited for content and length.


Journal Reference:

  1. C. Schlenoff, B. Weiss and M. Steves. A detailed discussion of lessons learned in evaluating emerging and advanced military technologies. International Journal of Intelligent Control and Systems, 2011; (forthcoming)

Cite This Page:

National Institute of Standards and Technology (NIST). "Researchers share useful lessons learned in evaluating emerging technologies." ScienceDaily. ScienceDaily, 27 June 2011. <www.sciencedaily.com/releases/2011/06/110627123042.htm>.
National Institute of Standards and Technology (NIST). (2011, June 27). Researchers share useful lessons learned in evaluating emerging technologies. ScienceDaily. Retrieved September 2, 2014 from www.sciencedaily.com/releases/2011/06/110627123042.htm
National Institute of Standards and Technology (NIST). "Researchers share useful lessons learned in evaluating emerging technologies." ScienceDaily. www.sciencedaily.com/releases/2011/06/110627123042.htm (accessed September 2, 2014).

Share This




More Science & Society News

Tuesday, September 2, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Get A Mortgage, Receive A Cat — Only In Russia

Get A Mortgage, Receive A Cat — Only In Russia

Newsy (Sep. 2, 2014) The incentive is in keeping with a Russian superstition that it's good luck for a cat to be the first to cross the threshold of a new home. Video provided by Newsy
Powered by NewsLook.com
Raw: Thousands of Fish Dead in Mexico Lake

Raw: Thousands of Fish Dead in Mexico Lake

AP (Sep. 2, 2014) Over 53 tons of rotting fish have been removed from Lake Cajititlan in western Jalisco state. Authorities say that the thousands of fish did not die of natural causes. (Sep. 2) Video provided by AP
Powered by NewsLook.com
U.N. Says Ebola Travel Restrictions Will Cause Food Shortage

U.N. Says Ebola Travel Restrictions Will Cause Food Shortage

Newsy (Sep. 2, 2014) The U.N. says the problem is two-fold — quarantine zones and travel restrictions are limiting the movement of both people and food. Video provided by Newsy
Powered by NewsLook.com
Halliburton Reaches $1B Gulf Spill Settlement

Halliburton Reaches $1B Gulf Spill Settlement

AP (Sep. 2, 2014) Halliburton's agreement to pay more than $1 billion to settle numerous claims involving the 2010 BP oil spill could be a way to diminish years of costly litigation. A federal judge still has to approve the settlement. (Sept. 2) Video provided by AP
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins