Featured Research

from universities, journals, and other organizations

Crowd workers are not online Shakespeares, but research shows they can write

Date:
February 2, 2011
Source:
Carnegie Mellon University
Summary:
Writing can be a solitary, intellectual pursuit, but researchers have shown that the task of writing an informational article also can be accomplished by dozens of people working independently online.

Writing can be a solitary, intellectual pursuit, but researchers at Carnegie Mellon University have shown that the task of writing an informational article also can be accomplished by dozens of people working independently online.

Related Articles


Each person in the CMU experiments completed just a sliver of the work of preparing an article, such as preparing an outline, gathering facts or assembling facts into simple prose. The "authors" never even spoke with each other. But the research team led by Aniket Kittur, assistant professor in CMU's Human-Computer Interaction Institute (HCII), found that the crowdsourced articles compared favorably with articles written by a single author and with Simple English Wikipedia entries.

"This is exciting because collaborative crowdsourcing could change the future of work," Kittur said. "We foresee a day when it will be possible to tap into hundreds of thousands or millions of workers around the globe to accomplish creative work on an unprecedented scale."

Kittur, along with Robert Kraut, professor of human-computer interaction, and Boris Smus, a student in HCII's joint master's degree program with the University of Madeira, have created a framework called CrowdForge that breaks down complex tasks into simple, independent micro-tasks that can be completed rapidly and cheaply. Their technical paper is available online at http://reports-archive.adm.cs.cmu.edu/anon/hcii/abstracts/11-100.html.

Jim Giles and MacGregor Campbell, San Francisco-based science journalists, have created a blog, www.mybossisarobot.com, that will explore the use of CrowdForge for preparing science news articles based on research reports.

Crowdsourcing has become a powerful mechanism for accomplishing work online. Millions of volunteers have performed tasks such as cataloging Martian landforms (http://beamartian.jpl.nasa.gov) and translating text into machine-readable form (http://recaptcha.com).

In the Carnegie Mellon experiments, crowdsourced work was performed through Amazon's Mechanical Turk (MTurk), an online marketplace for work. Employers can post simple, self-contained tasks on MTurk that workers, or "turkers," complete in return for a small fee, usually a few cents. Typical tasks include identifying objects in photos, writing product descriptions and transcribing audio recordings.

"But much of the work required by real-world organizations requires more time, cognitive effort and coordination among co-workers than is typical of these crowdsourcing efforts," Kittur said. Most turkers, for instance, refuse long, complex tasks because they are paid so little in return.

To accomplish these complex tasks, the CMU researchers approached the crowdsourcing market as if it was a distributed computing system, like the large computer systems used for Web searches. In a distributed computing system, computations are divided up in such a way that smaller chunks can be solved simultaneously by large numbers of processors and failures by individual processors won't undermine the entire process. Google, for instance, uses a framework called MapReduce in which queries are divided, or mapped, into sub-problems that can be solved simultaneously by numerous computers. The results of the computations then are combined, or reduced, to answer the query.

The framework developed by the CMU researchers, called CrowdForge, likewise divides up complex tasks so that many individuals can complete parts of the overall task and then provides a means for coordinating, combining and evaluating their work.

To prepare a brief encyclopedia article, for instance, CrowdForge would assign several people the task of writing an outline; as a quality control measure, a second set of workers might be tasked with voting for the best outline, or combining the best parts of each outline into a master outline. Subsequent sub-tasks might include collecting one fact for a topic in the outline. Finally, a worker might be given the task of taking several of the facts collected for a topic and turning them into a paragraph, or combining several paragraphs in proper order for an article.

In preparing five such articles on New York City, this method required an average of 36 sub-tasks for each article, at an average cost of $3.26. The articles averaged 658 words. The researchers then paid eight individuals $3.05 each to produce short articles on the same subjects; the average length was 393 words. When 15 people compared the articles, they rated the group-written articles of higher quality than those produced by individuals and about the same as a Wikipedia entry on the topic. The variability -- the range from the best to the worst article -- was lower for the crowdsourced articles.

"We were surprised at how well CrowdForge worked," Kittur said. "Admittedly, none of these articles is going to win any awards. But the ratings weren't bad considering that the work of dozens of people had to be coordinated to produce these pieces."

Kittur said the significance of CrowdForge is that it shows crowdsourcing of creative work is feasible, not that it can drive down the cost of articles. "We used MTurk as a source of workers," he noted, "but other users might tap into writers and researchers within an organization or into an existing network of freelancers."

This work was supported in part by grants from the National Science Foundation. More information is available on the CrowdForge project page, http://borismus.com/crowdforge.


Story Source:

The above story is based on materials provided by Carnegie Mellon University. Note: Materials may be edited for content and length.


Cite This Page:

Carnegie Mellon University. "Crowd workers are not online Shakespeares, but research shows they can write." ScienceDaily. ScienceDaily, 2 February 2011. <www.sciencedaily.com/releases/2011/02/110202152056.htm>.
Carnegie Mellon University. (2011, February 2). Crowd workers are not online Shakespeares, but research shows they can write. ScienceDaily. Retrieved October 26, 2014 from www.sciencedaily.com/releases/2011/02/110202152056.htm
Carnegie Mellon University. "Crowd workers are not online Shakespeares, but research shows they can write." ScienceDaily. www.sciencedaily.com/releases/2011/02/110202152056.htm (accessed October 26, 2014).

Share This



More Science & Society News

Sunday, October 26, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Texas Nurse Nina Pham Cured of Ebola

Texas Nurse Nina Pham Cured of Ebola

AFP (Oct. 25, 2014) An American nurse who contracted Ebola while caring for a Liberian patient in Texas has been declared free of the virus and will leave the hospital. Duration: 01:01 Video provided by AFP
Powered by NewsLook.com
Ebola Protective Suits Being Made in China

Ebola Protective Suits Being Made in China

AFP (Oct. 24, 2014) A factory in China is busy making Ebola protective suits for healthcare workers and others fighting the spread of the virus. Duration: 00:38 Video provided by AFP
Powered by NewsLook.com
WHO: Millions of Ebola Vaccine Doses by 2015

WHO: Millions of Ebola Vaccine Doses by 2015

AP (Oct. 24, 2014) The World Health Organization said on Friday that millions of doses of two experimental Ebola vaccines could be ready for use in 2015 and five more experimental vaccines would start being tested in March. (Oct. 24) Video provided by AP
Powered by NewsLook.com
Microsoft Riding High On Strong Surface, Cloud Performance

Microsoft Riding High On Strong Surface, Cloud Performance

Newsy (Oct. 24, 2014) Microsoft's Q3 earnings showed its tablets and cloud services are really hitting their stride. Video provided by Newsy
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:

Strange & Offbeat Stories


Science & Society

Business & Industry

Education & Learning

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins