A study comparing different statistical methods used to remove the effects of selection bias in observational studies finds that results may vary and caution may be warranted when interpreting findings of studies using certain methods, according to an article in the January 17 issue of JAMA.
With financial, practical, and ethical challenges involved in undertaking randomized clinical trials (RCTs), investigators often use observational data to compare the outcomes of different therapies, to guide policy statements and clinical protocols, and in generalizing results to the community. However, these comparisons may be biased due to important baseline differences in prognostic factors among patients, often as a result of unobserved treatment selection biases, according to background information in the article.
Therese A. Stukel, Ph.D., of the Institute for Clinical Evaluative Sciences, Toronto, and colleagues compared four analytic methods applied to the same data to determine if the estimated benefit from invasive therapy depends on the statistical method used to adjust for overt (measured) and hidden (unmeasured) bias. Methods included multivariable model risk adjustment, propensity score risk adjustment, and propensity-based matching, which control for overt bias, and instrumental variable analysis, which is a method designed to control for hidden bias as well.
The study included 122,124 patients who were elderly (age 65-84 years), receiving Medicare, and hospitalized with acute myocardial infarction (AMI; heart attack) in 1994-1995, and who were eligible for cardiac catheterization. Patients who received cardiac catheterization (n = 73,238) were younger and had lower AMI severity than those who did not. Baseline chart reviews were taken from the Cooperative Cardiovascular Project and linked to Medicare health administrative data to provide a set of prognostic variables. Patients were followed up for 7 years through December 2001, to assess the association between long-term survival and cardiac catheterization within 30 days of hospital admission.
The researchers found that, even after accounting for prognostic variables, cardiac catheterization was associated with an approximate 50 percent relative decrease in death rate, using standard risk-adjustment methods such as multivariable model risk adjustment, propensity score risk adjustment, or propensity-based matching. Using regional catheterization rate as an instrument, the instrumental variable analysis showed a 16 percent relative decrease in the death rate. The survival benefits of routine invasive care from randomized clinical trials are between 8 percent and 21 percent.
"Within a large observational data set, the estimated association of invasive cardiac treatment with long-term mortality is sensitive to the analytic method used," the authors write.
"Randomized clinical trials cannot be undertaken in all situations in which evidence is needed to guide care. Well-designed observational studies are still needed to assess population effectiveness and to extend results to a general population setting. Our study serves as a cautionary note regarding their analysis and interpretation. First, propensity scores and propensity-based matching have the same limitations as multivariable risk adjustment model methods, and are no more likely to remove bias due to unmeasured confounding when strong selection bias exists. Second, instrumental variable analyses may remove both overt and hidden biases but are more suited to answer policy questions than to provide insight into a specific clinical question for a specific patient. Caution is advised regarding clinical protocols and policy statements for invasive care based on expected mortality benefits derived from traditional multivariable modeling and propensity score risk adjustment of observational studies," the researchers conclude.
Cite This Page: