Featured Research

from universities, journals, and other organizations

Crowd workers are not online Shakespeares, but research shows they can write

Date:
February 2, 2011
Source:
Carnegie Mellon University
Summary:
Writing can be a solitary, intellectual pursuit, but researchers have shown that the task of writing an informational article also can be accomplished by dozens of people working independently online.

Writing can be a solitary, intellectual pursuit, but researchers at Carnegie Mellon University have shown that the task of writing an informational article also can be accomplished by dozens of people working independently online.

Each person in the CMU experiments completed just a sliver of the work of preparing an article, such as preparing an outline, gathering facts or assembling facts into simple prose. The "authors" never even spoke with each other. But the research team led by Aniket Kittur, assistant professor in CMU's Human-Computer Interaction Institute (HCII), found that the crowdsourced articles compared favorably with articles written by a single author and with Simple English Wikipedia entries.

"This is exciting because collaborative crowdsourcing could change the future of work," Kittur said. "We foresee a day when it will be possible to tap into hundreds of thousands or millions of workers around the globe to accomplish creative work on an unprecedented scale."

Kittur, along with Robert Kraut, professor of human-computer interaction, and Boris Smus, a student in HCII's joint master's degree program with the University of Madeira, have created a framework called CrowdForge that breaks down complex tasks into simple, independent micro-tasks that can be completed rapidly and cheaply. Their technical paper is available online at http://reports-archive.adm.cs.cmu.edu/anon/hcii/abstracts/11-100.html.

Jim Giles and MacGregor Campbell, San Francisco-based science journalists, have created a blog, www.mybossisarobot.com, that will explore the use of CrowdForge for preparing science news articles based on research reports.

Crowdsourcing has become a powerful mechanism for accomplishing work online. Millions of volunteers have performed tasks such as cataloging Martian landforms (http://beamartian.jpl.nasa.gov) and translating text into machine-readable form (http://recaptcha.com).

In the Carnegie Mellon experiments, crowdsourced work was performed through Amazon's Mechanical Turk (MTurk), an online marketplace for work. Employers can post simple, self-contained tasks on MTurk that workers, or "turkers," complete in return for a small fee, usually a few cents. Typical tasks include identifying objects in photos, writing product descriptions and transcribing audio recordings.

"But much of the work required by real-world organizations requires more time, cognitive effort and coordination among co-workers than is typical of these crowdsourcing efforts," Kittur said. Most turkers, for instance, refuse long, complex tasks because they are paid so little in return.

To accomplish these complex tasks, the CMU researchers approached the crowdsourcing market as if it was a distributed computing system, like the large computer systems used for Web searches. In a distributed computing system, computations are divided up in such a way that smaller chunks can be solved simultaneously by large numbers of processors and failures by individual processors won't undermine the entire process. Google, for instance, uses a framework called MapReduce in which queries are divided, or mapped, into sub-problems that can be solved simultaneously by numerous computers. The results of the computations then are combined, or reduced, to answer the query.

The framework developed by the CMU researchers, called CrowdForge, likewise divides up complex tasks so that many individuals can complete parts of the overall task and then provides a means for coordinating, combining and evaluating their work.

To prepare a brief encyclopedia article, for instance, CrowdForge would assign several people the task of writing an outline; as a quality control measure, a second set of workers might be tasked with voting for the best outline, or combining the best parts of each outline into a master outline. Subsequent sub-tasks might include collecting one fact for a topic in the outline. Finally, a worker might be given the task of taking several of the facts collected for a topic and turning them into a paragraph, or combining several paragraphs in proper order for an article.

In preparing five such articles on New York City, this method required an average of 36 sub-tasks for each article, at an average cost of $3.26. The articles averaged 658 words. The researchers then paid eight individuals $3.05 each to produce short articles on the same subjects; the average length was 393 words. When 15 people compared the articles, they rated the group-written articles of higher quality than those produced by individuals and about the same as a Wikipedia entry on the topic. The variability -- the range from the best to the worst article -- was lower for the crowdsourced articles.

"We were surprised at how well CrowdForge worked," Kittur said. "Admittedly, none of these articles is going to win any awards. But the ratings weren't bad considering that the work of dozens of people had to be coordinated to produce these pieces."

Kittur said the significance of CrowdForge is that it shows crowdsourcing of creative work is feasible, not that it can drive down the cost of articles. "We used MTurk as a source of workers," he noted, "but other users might tap into writers and researchers within an organization or into an existing network of freelancers."

This work was supported in part by grants from the National Science Foundation. More information is available on the CrowdForge project page, http://borismus.com/crowdforge.


Story Source:

The above story is based on materials provided by Carnegie Mellon University. Note: Materials may be edited for content and length.


Cite This Page:

Carnegie Mellon University. "Crowd workers are not online Shakespeares, but research shows they can write." ScienceDaily. ScienceDaily, 2 February 2011. <www.sciencedaily.com/releases/2011/02/110202152056.htm>.
Carnegie Mellon University. (2011, February 2). Crowd workers are not online Shakespeares, but research shows they can write. ScienceDaily. Retrieved April 17, 2014 from www.sciencedaily.com/releases/2011/02/110202152056.htm
Carnegie Mellon University. "Crowd workers are not online Shakespeares, but research shows they can write." ScienceDaily. www.sciencedaily.com/releases/2011/02/110202152056.htm (accessed April 17, 2014).

Share This



More Science & Society News

Thursday, April 17, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Small Reactors Could Be Future of Nuclear Energy

Small Reactors Could Be Future of Nuclear Energy

AP (Apr. 17, 2014) After the Fukushima nuclear disaster, the industry fell under intense scrutiny. Now, small underground nuclear power plants are being considered as the possible future of the nuclear energy. (April 17) Video provided by AP
Powered by NewsLook.com
Heartbleed Hack Leads To Arrest

Heartbleed Hack Leads To Arrest

Newsy (Apr. 17, 2014) A 19-year-old computer science student has been arrested in relation to a data breach of 900 social insurance numbers from Canada's revenue agency. Video provided by Newsy
Powered by NewsLook.com
Yahoo's Ousted COO Gets $58M Severance Package

Yahoo's Ousted COO Gets $58M Severance Package

Newsy (Apr. 17, 2014) According to SEC filings, Yahoo gave ousted COO Henrique de Castro a $58 million severance package. Video provided by Newsy
Powered by NewsLook.com
Are School Dress Codes Too Strict?

Are School Dress Codes Too Strict?

AP (Apr. 16, 2014) Pushing the limits on style and self-expression is a rite of passage for teens and even younger kids. How far should schools go with their dress codes? The courts have sided with schools in an era when school safety is paramount. (April 16) Video provided by AP
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins