Featured Research

from universities, journals, and other organizations

Computer Software Grades Essays Just As Well As People, Profs Announce

Date:
April 21, 1998
Source:
University Of Colorado At Boulder
Summary:
New computer software can grade the content of essay exams just as well as people and could be a major boon in assessing student performance, researchers at the University of Colorado at Boulder and New Mexico State University announced.

New computer software can grade the content of essay exams just as well as people and could be a major boon in assessing student performance, researchers at the University of Colorado at Boulder and New Mexico State University announced April 16.

Related Articles


"From sixth graders to first-year medical students we get consistently good results," said Thomas K. Landauer, a CU-Boulder psychology professor who has worked on the technology behind the program for 10 years. "It's ready."

The computer software, called Intelligent Essay Assessor, uses mathematical analysis to measure the quality of knowledge expressed in essays. It is the only automatic method for scoring the knowledge content of essays that has been extensively tested and published in peer-reviewed journals.

The system was developed by Landauer, Darrell Laham, a CU-Boulder doctoral student and Peter W. Foltz, an assistant professor of psychology at NMSU. They will discuss the system Thursday, April 16, during the annual meeting of the American Educational Research Association in San Diego.

"We are continually surprised at how well it works," said Landauer, who started on the project as director of cognitive science research at Bellcore.

The grading system has important implications for assessing student writing and helping students improve their writing, Foltz said. In one of his undergraduate psychology classes at NMSU last fall, Foltz tested a version of the program.

"Students submitted essays to a web page and received immediate feedback about the estimated grade for their essays, and suggestions about what was missing," Foltz said. "Students could revise their essays and resubmit them as many times as they wanted. The students' essays all improved with each revision."

Foltz also gave students the choice of having their essays graded by a human or by the computer. "They all chose to have the computer do the grading," he said.

Educators laud essay exams because they provide a better assessment of students' knowledge than other types of tests. A huge drawback is that the tests are time-consuming and difficult to grade fairly and accurately, particularly for large classes or nationally administered exams.

But computer-based evaluations of student writing are becoming increasingly feasible because of the growing numbers of students who write using computers. The researchers have applied for a patent on their software.

The new system requires a computer with about 20 times the memory of an ordinary PC to do the statistical analysis that it needs to "understand" essays. It uses Latent Semantic Analysis, a new type of artificial intelligence that is much like a neural network. "In a sense, it tries to mimic the function of the human brain," Laham said.

First the software program is "fed" information about a topic in the form of 50,000 to 10 million words from on-line textbooks or other sources. It learns from the text and then assigns a mathematical degree of similarity or "distance" between the meaning of each word and any other word. This allows students to use different words that mean the same thing and receive the same score. For example, they could use "physician" instead of "doctor."

The program then evaluates essays in two primary ways. The first is for a teacher or professor to grade enough essays to provide a good statistical sample and then use the software to grade the remainder.

"It takes the combination of words in the student essay and computes its similarity to the combination of words in the comparison essays," Laham said. The student then receives the same grade as the human-graded essays to which it is most closely matched.

"The program has perfect consistency in grading -- an attribute that human graders almost never have," Laham said. "The system does not get bored, rushed, sleepy, impatient or forgetful." In one test, both the Intelligent Essay Assessor and faculty members graded essays from 500 psychology students at CU-Boulder. "The correlation between the two scores was very high -- it was the same correlation as if two humans were reading them," Landauer said.

The software only evaluates knowledge content and is not designed to grade stylistic considerations like grammar and spelling, researchers said. Existing programs already can do those functions.

A second Intelligent Essay Assessor method compares all the student essays to a single professor's or expert's essay, a so-called "gold standard." A third variation can tell students what important subject matter was missing from their essays and where to find it in the textbook.

Previous methods of automatic essay scoring simply counted words and then analyzed mechanics and aspects of grammatical style, the researchers said.

There is a strong correlation between students who write the most and students who write the best, researchers said. This is because students who know a lot write a lot.

The amount of content also counts in the Intelligent Essay Assessor, but it is measured by concepts, not by the number of words. The researchers recommend setting an essay word limit to eliminate length as a factor.

Because the system does not analyze surface form, it is possible that someone could include all the right words in an essay -- in random order -- and get a good grade, they said. The system will flag unusual essays for that and other reasons for a human to check. But the team discovered an even better safeguard while trying to fool the system.

"If you wrote a good essay and scrambled the words you would get a good grade," Landauer said. "But try to get the good words without writing a good essay!

"We've tried to write bad essays and get good grades and we can sometimes do it if we know the material really well. The easiest way to cheat this system is to study hard, know the material and write a good essay."


Story Source:

The above story is based on materials provided by University Of Colorado At Boulder. Note: Materials may be edited for content and length.


Cite This Page:

University Of Colorado At Boulder. "Computer Software Grades Essays Just As Well As People, Profs Announce." ScienceDaily. ScienceDaily, 21 April 1998. <www.sciencedaily.com/releases/1998/04/980421080525.htm>.
University Of Colorado At Boulder. (1998, April 21). Computer Software Grades Essays Just As Well As People, Profs Announce. ScienceDaily. Retrieved December 17, 2014 from www.sciencedaily.com/releases/1998/04/980421080525.htm
University Of Colorado At Boulder. "Computer Software Grades Essays Just As Well As People, Profs Announce." ScienceDaily. www.sciencedaily.com/releases/1998/04/980421080525.htm (accessed December 17, 2014).

Share This


More From ScienceDaily



More Computers & Math News

Wednesday, December 17, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Jaguar Unveils 360° Virtual Windshield Making Car Pillars Appear Transparent

Jaguar Unveils 360° Virtual Windshield Making Car Pillars Appear Transparent

Buzz60 (Dec. 17, 2014) — Jaguar unveils a virtual 360 degree windshield that may be the most futuristic automotive development yet. Jen Markham explains. Video provided by Buzz60
Powered by NewsLook.com
BlackBerry Launches Classic Smartphone

BlackBerry Launches Classic Smartphone

AP (Dec. 17, 2014) — BlackBerry is returning to its roots with a new smartphone called the Classic, featuring a traditional keyboard at a time when rival Apple and Android phones - and most smartphone customers - have embraced touch screens. (Dec. 17) Video provided by AP
Powered by NewsLook.com
The Future of Work, Skills & Careers in a Digital World-Dr. Tracy Wilen

The Future of Work, Skills & Careers in a Digital World-Dr. Tracy Wilen

Working Mother (Dec. 16, 2014) — 2014 Worklife Congress Video provided by Working Mother
Powered by NewsLook.com
Tech Companies Make Holiday Shopping Easier Than Ever

Tech Companies Make Holiday Shopping Easier Than Ever

Newsy (Dec. 16, 2014) — Innovative new services allow consumers to shop with their smartphones, split bills and even haggle. Video provided by Newsy
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
 
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:  

Breaking News:

Strange & Offbeat Stories

 

Space & Time

Matter & Energy

Computers & Math

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:  

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile iPhone Android Web
Follow Facebook Twitter Google+
Subscribe RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins