Featured Research

from universities, journals, and other organizations

Addressing 'mischievous responders' would increase validity of adolescent research

Date:
May 15, 2014
Source:
American Educational Research Association (AERA)
Summary:
“Mischievous responders” play the game of intentionally providing inaccurate answers on anonymous surveys, a widespread problem that can mislead research findings. However, new data analysis procedures may help minimize the impact of these “jokester youths.” By providing misleading responses that the responders think are funny, these people, even in small numbers, can lead researchers to wildly incorrect conclusions. In turn, the conclusions can lead to ineffective policymaking and may perpetuate negative stereotypes about marginalized groups.

"Mischievous responders" play the game of intentionally providing inaccurate answers on anonymous surveys, a widespread problem that can mislead research findings. However, new data analysis procedures may help minimize the impact of these "jokester youths," according to research published online today in Educational Researcher, a peer-reviewed journal of the American Educational Research Association (AERA).

"Inaccurate Estimation of Disparities Due to Mischievous Responders: Several Suggestions to Assess Conclusions," by Joseph P. Robinson-Cimpian of the University of Illinois at Urbana-Champaign, suggests that mischievous responders are present in commonly used data sets and that their presence may severely alter impressions of relative risk between groups of adolescents. The article introduces novel sensitivity-analysis procedures for investigating and reducing the bias that mischievous responders often introduce in research on adolescents.

Mischievous responders are youths who provide extreme and potentially untruthful responses to multiple questions on self-administered questionnaires (SAQs). By providing misleading responses that they think are funny, these responders, even in small numbers, can lead researchers to wildly incorrect conclusions. In turn, the conclusions can lead to ineffective policymaking and may perpetuate negative stereotypes about marginalized groups.

In his article, Robinson-Cimpian demonstrates that even a very small proportion of mischievous responders can lead to exaggerated risk estimates for lesbian, gay, bisexual, and questioning (LGBQ), transgender, and disabled youths, with regard to drug usage, suicidal thoughts, and school disengagement. For example, potentially mischievous responders caused LGBQ-heterosexual disparities in recent cocaine/crack use to inflate from 4 percentage points to 12 percentage points.

To identify mischievous responders, Robinson-Cimpian's method relies on detecting patterns of unusual answers provided by survey respondents.

"If we find that youths reporting to be gay are more likely than those reporting to be straight to say that they are blind and deaf and extremely tall and parenting multiple children all at the same time, then we might question whether the data are valid," said Robinson-Cimpian. "Just like these jokester youths think it's funny to say that they are gay and blind, they also think it's funny to say that they are suicidal, engage in sexually risky behavior, and take drugs. And this can dramatically affect our estimates of risk."

SAQs are often used to estimate the psychological and health disparities between groups, including adoptees and non-adoptees, sexual minorities and non-minorities, racial/ethnic minorities and non-minorities, and individuals with and without disabilities, among others. The anonymity often provided by SAQs makes them an important tool in adolescent research, but Robinson-Cimpian cautions that researchers must take steps to help ensure the data are valid.

"Within the past decade, much research has been criticized for failure to replicate and for exaggerated results," wrote Robinson-Cimpian in his article. He also said, "If we want sound research and policy, we need to have sound data. The procedures introduced here have broad relevance to research and can be widely, and easily, implemented."

The proposed approach does not require additional data collection, which makes it more practical, cost-effective, and applicable to already collected data. Robinson-Cimpian's technique is unique in that it can be used in situations where it is impossible or unethical to verify information by comparing adolescents' responses with those of their parents or by conducting in-person follow-up interviews with the respondents.

The four-step sensitivity analysis proposed by Robinson-Cimpian requires first identifying youths who provide high numbers of low-frequency responses and then comparing estimated disparities when including and excluding these youths.

While it is possible that some respondents will be mistakenly identified as mischievous, or that some mischievous respondents will not provide enough low-frequency responses to be identified as mischievous, there is great potential for both scientific and practical damage when no analysis is performed to assess data validity and ensure robust results, according to Robinson-Cimpian. This includes perpetuated stigma against minority groups.

For instance, this type of sensitivity analysis may have prevented well-intentioned research from being used by groups that claim LGBTQ identification leads to substance abuse and other risky behaviors, and thus advocate against using school resources to help LGBTQ youths, wrote Robinson-Cimpian in his article.


Story Source:

The above story is based on materials provided by American Educational Research Association (AERA). Note: Materials may be edited for content and length.


Journal Reference:

  1. J. P. Robinson-Cimpian. Inaccurate Estimation of Disparities Due to Mischievous Responders: Several Suggestions to Assess Conclusions. Educational Researcher, 2014; DOI: 10.3102/0013189X14534297

Cite This Page:

American Educational Research Association (AERA). "Addressing 'mischievous responders' would increase validity of adolescent research." ScienceDaily. ScienceDaily, 15 May 2014. <www.sciencedaily.com/releases/2014/05/140515095156.htm>.
American Educational Research Association (AERA). (2014, May 15). Addressing 'mischievous responders' would increase validity of adolescent research. ScienceDaily. Retrieved July 22, 2014 from www.sciencedaily.com/releases/2014/05/140515095156.htm
American Educational Research Association (AERA). "Addressing 'mischievous responders' would increase validity of adolescent research." ScienceDaily. www.sciencedaily.com/releases/2014/05/140515095156.htm (accessed July 22, 2014).

Share This




More Mind & Brain News

Tuesday, July 22, 2014

Featured Research

from universities, journals, and other organizations


Featured Videos

from AP, Reuters, AFP, and other news services

Do Obese Women Have 'Food Learning Impairment'?

Do Obese Women Have 'Food Learning Impairment'?

Newsy (July 18, 2014) Yale researchers tested 135 men and women, and it was only obese women who were deemed to have "impaired associative learning." Video provided by Newsy
Powered by NewsLook.com
Does Mixing Alcohol and Energy Drinks Boost Urge To Drink?

Does Mixing Alcohol and Energy Drinks Boost Urge To Drink?

Newsy (July 18, 2014) A new study suggests that mixing alcohol with energy drinks makes you want to keep the party going. Video provided by Newsy
Powered by NewsLook.com
Pot Cooking Class Teaches Responsible Eating

Pot Cooking Class Teaches Responsible Eating

AP (July 18, 2014) Following the nationwide trend of eased restrictions on marijuana use, pot edibles are growing in popularity. One Boston-area cooking class is teaching people how to eat pot responsibly. (July 18) Video provided by AP
Powered by NewsLook.com
Understanding D.C.'s New Pot Laws

Understanding D.C.'s New Pot Laws

Newsy (July 17, 2014) Washington D.C.'s new laws decriminalizing small amount of marijuana went into effect Thursday. Here's how they work. Video provided by Newsy
Powered by NewsLook.com

Search ScienceDaily

Number of stories in archives: 140,361

Find with keyword(s):
Enter a keyword or phrase to search ScienceDaily for related topics and research stories.

Save/Print:
Share:

Breaking News:
from the past week

In Other News

... from NewsDaily.com

Science News

Health News

Environment News

Technology News



Save/Print:
Share:

Free Subscriptions


Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

Get Social & Mobile


Keep up to date with the latest news from ScienceDaily via social networks and mobile apps:

Have Feedback?


Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?
Mobile: iPhone Android Web
Follow: Facebook Twitter Google+
Subscribe: RSS Feeds Email Newsletters
Latest Headlines Health & Medicine Mind & Brain Space & Time Matter & Energy Computers & Math Plants & Animals Earth & Climate Fossils & Ruins