Featured Research

from universities, journals, and other organizations

Addressing 'mischievous responders' would increase validity of adolescent research

Date:
May 15, 2014
Source:
American Educational Research Association (AERA)
Summary:
“Mischievous responders” play the game of intentionally providing inaccurate answers on anonymous surveys, a widespread problem that can mislead research findings. However, new data analysis procedures may help minimize the impact of these “jokester youths.” By providing misleading responses that the responders think are funny, these people, even in small numbers, can lead researchers to wildly incorrect conclusions. In turn, the conclusions can lead to ineffective policymaking and may perpetuate negative stereotypes about marginalized groups.

"Mischievous responders" play the game of intentionally providing inaccurate answers on anonymous surveys, a widespread problem that can mislead research findings. However, new data analysis procedures may help minimize the impact of these "jokester youths," according to research published online today in Educational Researcher, a peer-reviewed journal of the American Educational Research Association (AERA).

"Inaccurate Estimation of Disparities Due to Mischievous Responders: Several Suggestions to Assess Conclusions," by Joseph P. Robinson-Cimpian of the University of Illinois at Urbana-Champaign, suggests that mischievous responders are present in commonly used data sets and that their presence may severely alter impressions of relative risk between groups of adolescents. The article introduces novel sensitivity-analysis procedures for investigating and reducing the bias that mischievous responders often introduce in research on adolescents.

Mischievous responders are youths who provide extreme and potentially untruthful responses to multiple questions on self-administered questionnaires (SAQs). By providing misleading responses that they think are funny, these responders, even in small numbers, can lead researchers to wildly incorrect conclusions. In turn, the conclusions can lead to ineffective policymaking and may perpetuate negative stereotypes about marginalized groups.

In his article, Robinson-Cimpian demonstrates that even a very small proportion of mischievous responders can lead to exaggerated risk estimates for lesbian, gay, bisexual, and questioning (LGBQ), transgender, and disabled youths, with regard to drug usage, suicidal thoughts, and school disengagement. For example, potentially mischievous responders caused LGBQ-heterosexual disparities in recent cocaine/crack use to inflate from 4 percentage points to 12 percentage points.

To identify mischievous responders, Robinson-Cimpian's method relies on detecting patterns of unusual answers provided by survey respondents.

"If we find that youths reporting to be gay are more likely than those reporting to be straight to say that they are blind and deaf and extremely tall and parenting multiple children all at the same time, then we might question whether the data are valid," said Robinson-Cimpian. "Just like these jokester youths think it's funny to say that they are gay and blind, they also think it's funny to say that they are suicidal, engage in sexually risky behavior, and take drugs. And this can dramatically affect our estimates of risk."

SAQs are often used to estimate the psychological and health disparities between groups, including adoptees and non-adoptees, sexual minorities and non-minorities, racial/ethnic minorities and non-minorities, and individuals with and without disabilities, among others. The anonymity often provided by SAQs makes them an important tool in adolescent research, but Robinson-Cimpian cautions that researchers must take steps to help ensure the data are valid.

"Within the past decade, much research has been criticized for failure to replicate and for exaggerated results," wrote Robinson-Cimpian in his article. He also said, "If we want sound research and policy, we need to have sound data. The procedures introduced here have broad relevance to research and can be widely, and easily, implemented."

The proposed approach does not require additional data collection, which makes it more practical, cost-effective, and applicable to already collected data. Robinson-Cimpian's technique is unique in that it can be used in situations where it is impossible or unethical to verify information by comparing adolescents' responses with those of their parents or by conducting in-person follow-up interviews with the respondents.

The four-step sensitivity analysis proposed by Robinson-Cimpian requires first identifying youths who provide high numbers of low-frequency responses and then comparing estimated disparities when including and excluding these youths.

While it is possible that some respondents will be mistakenly identified as mischievous, or that some mischievous respondents will not provide enough low-frequency responses to be identified as mischievous, there is great potential for both scientific and practical damage when no analysis is performed to assess data validity and ensure robust results, according to Robinson-Cimpian. This includes perpetuated stigma against minority groups.

For instance, this type of sensitivity analysis may have prevented well-intentioned research from being used by groups that claim LGBTQ identification leads to substance abuse and other risky behaviors, and thus advocate against using school resources to help LGBTQ youths, wrote Robinson-Cimpian in his article.


Story Source:

The above story is based on materials provided by American Educational Research Association (AERA). Note: Materials may be edited for content and length.


Journal Reference:

  1. J. P. Robinson-Cimpian. Inaccurate Estimation of Disparities Due to Mischievous Responders: Several Suggestions to Assess Conclusions. Educational Researcher, 2014; DOI: 10.3102/0013189X14534297

Cite This Page:

American Educational Research Association (AERA). "Addressing 'mischievous responders' would increase validity of adolescent research." ScienceDaily. ScienceDaily, 15 May 2014. <www.sciencedaily.com/releases/2014/05/140515095156.htm>.
American Educational Research Association (AERA). (2014, May 15). Addressing 'mischievous responders' would increase validity of adolescent research. ScienceDaily. Retrieved September 1, 2014 from www.sciencedaily.com/releases/2014/05/140515095156.htm
American Educational Research Association (AERA). "Addressing 'mischievous responders' would increase validity of adolescent research." ScienceDaily. www.sciencedaily.com/releases/2014/05/140515095156.htm (accessed September 1, 2014).

Share This




More Mind & Brain News

Monday, September 1, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Coffee Then Napping: The (New) Key To Alertness

Coffee Then Napping: The (New) Key To Alertness

Newsy (Aug. 30, 2014) Researchers say having a cup of coffee then taking a nap is more effective than a nap or coffee alone. Video provided by Newsy
Powered by NewsLook.com
Young Entrepreneurs Get $100,000, If They Quit School

Young Entrepreneurs Get $100,000, If They Quit School

AFP (Aug. 29, 2014) Twenty college-age students are getting 100,000 dollars from a Silicon Valley leader and a chance to live in San Francisco in order to work on the start-up project of their dreams, but they have to quit school first. Duration: 02:20 Video provided by AFP
Powered by NewsLook.com
Baby Babbling Might Lead To Faster Language Development

Baby Babbling Might Lead To Faster Language Development

Newsy (Aug. 29, 2014) A new study suggests babies develop language skills more quickly if their parents imitate the babies' sounds and expressions and talk to them often. Video provided by Newsy
Powered by NewsLook.com
Electrical Stimulation Boosts Brain Function, Study Says

Electrical Stimulation Boosts Brain Function, Study Says

Newsy (Aug. 29, 2014) Researchers found an improvement in memory and learning function in subjects who received electric pulses to their brains. Video provided by Newsy
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins