Featured Research

from universities, journals, and other organizations

Addressing 'mischievous responders' would increase validity of adolescent research

Date:
May 15, 2014
Source:
American Educational Research Association (AERA)
Summary:
“Mischievous responders” play the game of intentionally providing inaccurate answers on anonymous surveys, a widespread problem that can mislead research findings. However, new data analysis procedures may help minimize the impact of these “jokester youths.” By providing misleading responses that the responders think are funny, these people, even in small numbers, can lead researchers to wildly incorrect conclusions. In turn, the conclusions can lead to ineffective policymaking and may perpetuate negative stereotypes about marginalized groups.

"Mischievous responders" play the game of intentionally providing inaccurate answers on anonymous surveys, a widespread problem that can mislead research findings. However, new data analysis procedures may help minimize the impact of these "jokester youths," according to research published online today in Educational Researcher, a peer-reviewed journal of the American Educational Research Association (AERA).

"Inaccurate Estimation of Disparities Due to Mischievous Responders: Several Suggestions to Assess Conclusions," by Joseph P. Robinson-Cimpian of the University of Illinois at Urbana-Champaign, suggests that mischievous responders are present in commonly used data sets and that their presence may severely alter impressions of relative risk between groups of adolescents. The article introduces novel sensitivity-analysis procedures for investigating and reducing the bias that mischievous responders often introduce in research on adolescents.

Mischievous responders are youths who provide extreme and potentially untruthful responses to multiple questions on self-administered questionnaires (SAQs). By providing misleading responses that they think are funny, these responders, even in small numbers, can lead researchers to wildly incorrect conclusions. In turn, the conclusions can lead to ineffective policymaking and may perpetuate negative stereotypes about marginalized groups.

In his article, Robinson-Cimpian demonstrates that even a very small proportion of mischievous responders can lead to exaggerated risk estimates for lesbian, gay, bisexual, and questioning (LGBQ), transgender, and disabled youths, with regard to drug usage, suicidal thoughts, and school disengagement. For example, potentially mischievous responders caused LGBQ-heterosexual disparities in recent cocaine/crack use to inflate from 4 percentage points to 12 percentage points.

To identify mischievous responders, Robinson-Cimpian's method relies on detecting patterns of unusual answers provided by survey respondents.

"If we find that youths reporting to be gay are more likely than those reporting to be straight to say that they are blind and deaf and extremely tall and parenting multiple children all at the same time, then we might question whether the data are valid," said Robinson-Cimpian. "Just like these jokester youths think it's funny to say that they are gay and blind, they also think it's funny to say that they are suicidal, engage in sexually risky behavior, and take drugs. And this can dramatically affect our estimates of risk."

SAQs are often used to estimate the psychological and health disparities between groups, including adoptees and non-adoptees, sexual minorities and non-minorities, racial/ethnic minorities and non-minorities, and individuals with and without disabilities, among others. The anonymity often provided by SAQs makes them an important tool in adolescent research, but Robinson-Cimpian cautions that researchers must take steps to help ensure the data are valid.

"Within the past decade, much research has been criticized for failure to replicate and for exaggerated results," wrote Robinson-Cimpian in his article. He also said, "If we want sound research and policy, we need to have sound data. The procedures introduced here have broad relevance to research and can be widely, and easily, implemented."

The proposed approach does not require additional data collection, which makes it more practical, cost-effective, and applicable to already collected data. Robinson-Cimpian's technique is unique in that it can be used in situations where it is impossible or unethical to verify information by comparing adolescents' responses with those of their parents or by conducting in-person follow-up interviews with the respondents.

The four-step sensitivity analysis proposed by Robinson-Cimpian requires first identifying youths who provide high numbers of low-frequency responses and then comparing estimated disparities when including and excluding these youths.

While it is possible that some respondents will be mistakenly identified as mischievous, or that some mischievous respondents will not provide enough low-frequency responses to be identified as mischievous, there is great potential for both scientific and practical damage when no analysis is performed to assess data validity and ensure robust results, according to Robinson-Cimpian. This includes perpetuated stigma against minority groups.

For instance, this type of sensitivity analysis may have prevented well-intentioned research from being used by groups that claim LGBTQ identification leads to substance abuse and other risky behaviors, and thus advocate against using school resources to help LGBTQ youths, wrote Robinson-Cimpian in his article.


Story Source:

The above story is based on materials provided by American Educational Research Association (AERA). Note: Materials may be edited for content and length.


Journal Reference:

  1. J. P. Robinson-Cimpian. Inaccurate Estimation of Disparities Due to Mischievous Responders: Several Suggestions to Assess Conclusions. Educational Researcher, 2014; DOI: 10.3102/0013189X14534297

Cite This Page:

American Educational Research Association (AERA). "Addressing 'mischievous responders' would increase validity of adolescent research." ScienceDaily. ScienceDaily, 15 May 2014. <www.sciencedaily.com/releases/2014/05/140515095156.htm>.
American Educational Research Association (AERA). (2014, May 15). Addressing 'mischievous responders' would increase validity of adolescent research. ScienceDaily. Retrieved September 30, 2014 from www.sciencedaily.com/releases/2014/05/140515095156.htm
American Educational Research Association (AERA). "Addressing 'mischievous responders' would increase validity of adolescent research." ScienceDaily. www.sciencedaily.com/releases/2014/05/140515095156.htm (accessed September 30, 2014).

Share This



More Mind & Brain News

Tuesday, September 30, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Do Video Games Trump Brain Training For Cognitive Boosts?

Do Video Games Trump Brain Training For Cognitive Boosts?

Newsy (Sep. 29, 2014) More and more studies are showing positive benefits to playing video games, but the jury is still out on brain training programs. Video provided by Newsy
Powered by NewsLook.com
Your Spouse's Personality May Influence Your Earnings

Your Spouse's Personality May Influence Your Earnings

Newsy (Sep. 26, 2014) Research from Washington University suggest people with conscientious spouses have greater career success. Video provided by Newsy
Powered by NewsLook.com
Can A Blood Test Predict Psychosis Risk?

Can A Blood Test Predict Psychosis Risk?

Newsy (Sep. 26, 2014) Researchers say certain markers in the blood can predict risk of psychosis later in the life. The test can aid in early treatment for the condition. Video provided by Newsy
Powered by NewsLook.com
Harpist Soothes Gorillas, Orangutans With Music

Harpist Soothes Gorillas, Orangutans With Music

AP (Sep. 25, 2014) Teri Tacheny, a harpist, has a loyal following of fans who appreciate her soothing music. Every month, gorillas, orangutans and monkeys amble down to hear her play at the Como Park Zoo in Minnesota. (Sept. 25) Video provided by AP
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:

Strange & Offbeat Stories


Health & Medicine

Mind & Brain

Living & Well

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins