Featured Research

from universities, journals, and other organizations

Researchers share useful lessons learned in evaluating emerging technologies

Date:
June 27, 2011
Source:
National Institute of Standards and Technology (NIST)
Summary:
A team from the U.S. National Institute of Standards and Technology shares critical "lessons learned" that can help businesses and others negotiate the promises and pitfalls encountered when pushing the technology envelope to enable new capabilities.

A NIST-designed framework for evaluating emerging technologies helped to focus and further efforts to develop two-way speech-translation systems to aid US soldiers and Marines serving in Iraq and Afghanistan. In this trial, a Marine and a speaker of Pashto, a native Afghani tongue, conversed using a translation system based on smart phone technology. The technology-development effort was sponsored by the Defense Advanced Research Projects Agency.
Credit: NIST

Most industry executives, military planners, research managers or venture capitalists charged with assessing the potential of an R&D project probably are familiar with the wry twist on Arthur C. Clarke's third law: "Any sufficiently advanced technology is indistinguishable from a rigged demo."

After serving for five years as independent evaluators of emerging military technologies nurtured by the Defense Advanced Research Projects Agency (DARPA), a team from the National Institute of Standards and Technology (NIST) shares critical "lessons learned" that can help businesses and others negotiate the promises and pitfalls encountered when pushing the technology envelope to enable new capabilities.

Writing in the International Journal of Intelligent Control and Systems, the NIST researchers also describe the evaluative framework they devised for judging the performance of a system and its components as well as the utility of the technology for the intended user. Called SCORE (System, Component, and Operationally Relevant Evaluations), the framework is a unified set of criteria and software tools for evaluating emerging technologies from different perspectives and levels of detail and at various stages of development.

SCORE was developed for evaluating so-called intelligent systems -- a fast growing category of technologies ranging from robots and unmanned vehicles to sensor networks, natural language processing devices and "smart" appliances. By definition, explains Craig Schlenoff, acting head of NIST's Systems Integration Group, "Intelligent systems can respond to conditions in an uncertain environment -- be it a battlefield, a factory floor, or an urban highway system -- in ways that help the technology accomplish its intended purpose."

Schlenoff and his colleagues used their SCORE approach to evaluate technologies as they progressed under two DARPA programs: ASSIST and TRANSTAC. In ASSIST, DARPA is funding efforts to instrument soldiers with wearable sensors -- video cameras, microphones, global positioning devices and more -- to continuously record activities while they are on a mission. TRANSTAC is driving the development of two-way speech-translation systems that enable speakers of different languages to communicate with each other in real-world situations, without an interpreter. By providing constructive feedback on system capabilities, the SCORE evaluative framework helps to drive innovation and performance improvements.

Several lessons learned recounted by the NIST team are aimed at maximizing the contributions of test subjects and the developers of technologies without biasing test results. "There is often a balancing act," they write, "between creating the evaluation environment in a way that shows the system in the best possible light vs. having an environment that is as realistic as possible."

They also discuss unavoidable trade-offs due to costs, logistics, or other factors. While evaluators and technology developers should never lose sight of their ultimate objective, the NIST researchers also advise the need for flexibility over time. As more is learned about the system and about user requirements, features may change and project goals may be modified, necessitating adjustments to the evaluation approach.

"The main lesson," Schlenoff explains, "is that the extra effort devoted to evaluation planning can have a huge effect on how successful the evaluation will be. Bad decisions made during the design can be difficult and costly to fix later on."

To learn more about SCORE, go to: http://www.nist.gov/el/isd/ks/score.cfm.


Story Source:

The above story is based on materials provided by National Institute of Standards and Technology (NIST). Note: Materials may be edited for content and length.


Journal Reference:

  1. C. Schlenoff, B. Weiss and M. Steves. A detailed discussion of lessons learned in evaluating emerging and advanced military technologies. International Journal of Intelligent Control and Systems, 2011; (forthcoming)

Cite This Page:

National Institute of Standards and Technology (NIST). "Researchers share useful lessons learned in evaluating emerging technologies." ScienceDaily. ScienceDaily, 27 June 2011. <www.sciencedaily.com/releases/2011/06/110627123042.htm>.
National Institute of Standards and Technology (NIST). (2011, June 27). Researchers share useful lessons learned in evaluating emerging technologies. ScienceDaily. Retrieved April 23, 2014 from www.sciencedaily.com/releases/2011/06/110627123042.htm
National Institute of Standards and Technology (NIST). "Researchers share useful lessons learned in evaluating emerging technologies." ScienceDaily. www.sciencedaily.com/releases/2011/06/110627123042.htm (accessed April 23, 2014).

Share This



More Science & Society News

Wednesday, April 23, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Nuclear-Level Asteroids Might Be More Common Than We Realize

Nuclear-Level Asteroids Might Be More Common Than We Realize

Newsy (Apr. 23, 2014) The B612 Foundation says asteroids strike Earth much more often than previously thought, and are hoping to build an early warning system. Video provided by Newsy
Powered by NewsLook.com
High Court to Hear Dispute of TV Over Internet

High Court to Hear Dispute of TV Over Internet

AP (Apr. 22, 2014) The future of Aereo, an online service that provides over-the-air TV channels, hinges on a battle with broadcasters that goes before the U.S. Supreme Court on Tuesday. (April 22) Video provided by AP
Powered by NewsLook.com
Is North Korea Planning Nuclear Test #4?

Is North Korea Planning Nuclear Test #4?

Newsy (Apr. 22, 2014) South Korean officials say North Korea is preparing to conduct another nuclear test, but is Pyongyang just bluffing this time? Video provided by Newsy
Powered by NewsLook.com
Study Says Most Crime Not Linked To Mental Illness

Study Says Most Crime Not Linked To Mental Illness

Newsy (Apr. 22, 2014) A new study finds most crimes committed by people with mental illness are not caused by symptoms of their illness or disorder. Video provided by Newsy
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



    Save/Print:
    Share:

    Free Subscriptions


    Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

    Get Social & Mobile


    Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

    Have Feedback?


    Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
    Mobile: iPhone Android Web
    Follow: Facebook Twitter Google+
    Subscribe: RSS Feeds Email Newsletters
    Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins