Just knowing isn't enough: Issuing hospital 'report cards' had no impact on surgical outcomes

The not-so-good news: Those "report cards" do not seem to be making things better for patients.

A new study published today in JAMA by a team from the University of Michigan shows no difference in surgical safety among 263 hospitals taking part in a major national quality effort, and 526 similar hospitals that weren't involved. The study analyzed data from 1,226,000 seniors enrolled in Medicare who had one of 11 major operations at those hospitals over a decade.

The initiative, called the American College of Surgeons National Surgical Quality Improvement Program or ACS-NSQIP, has been run since the early 2000s. Trained nurses at participating hospitals record data about every operation carefully, and send it to a secure central database. The ACS crunches the data from all ACS-NSQIP hospitals and shares quality reports, allowing hospitals and doctors see how their overall performance stacks up against others.

But this quality reporting, the authors find, is not enough alone to accelerate the pace of improvement in surgical safety -- nor cost savings.

That doesn't mean such efforts should stop, or aren't worthwhile, they hasten to add.

But it does point to the need for better efforts to ensure hospitals put the ACS-NSQIP data to good use, and work with other hospitals in their region to develop and share "best practices" that lead to improvements across hospitals. The ACS, and other large health care quality organizations such as the Michigan Surgical Quality Collaborative, have recently begun to emphasize the importance of collaboration.

"Although ACS-NSQIP hospitals are improving over time, so are other non-participating hospitals," says Nicholas Osborne, M.D., M.S., the study's lead author and a vascular surgeon at the U-M Health System's Frankel Cardiovascular Center. "Our study suggests that the ACS-NSQIP is a good start, but that reporting data back to hospitals is not enough. The 'drilling down' that is needed to improve quality using these reports is better suited for regional collaboratives."

Osborne notes that the analysis is the first to use a control group of hospitals to study the impact of ACS-NSQIP participation -- and that the team matched each ACS-NSQIP hospital with not one, but two control hospitals. Patients treated at the two types of hospitals were generally similar, though ACS-NSQIP hospitals were larger and did more operations and were more likely to be nonprofits or teaching hospitals.

The 11 types of operations analyzed were esophagectomy, pancreatic resection, colon resection, gastrectomy, liver resection, ventral hernia repair, cholecystectomy, appendectomy, abdominal aortic aneurysm repair, lower extremity bypass, and carotid endarterectomy.

The lack of an effect from ACS-NSQIP participation could be due to many things, the authors say, including hospitals may not have used the reports to improve care, or quality improvement efforts by hospitals using their data may have fallen short of affecting the four items the study evaluated. Many hospitals may not have the infrastructure needed to develop effective strategies to improve care. In addition, outside factors, such as reimbursement-driven efforts to improve safety, improvements in care across all hospitals or selective referral of patients to high-volume hospitals, could also have played a role in improving safety at all hospitals.

"Knowing where you perform poorly is the important first step," says Osborne. "But the next leap from measuring outcomes to improving outcomes is much more difficult."

Adds senior author Justin Dimick, M.D., MPH, "Better approaches for engaging surgeons, better systems for supporting them in change efforts, and better tools for helping them re-engineer care are clearly needed. Future national and regional quality improvement initiatives must be aimed at not only providing feedback to participants, but also providing an infrastructure for implementing change."

How the study was done

Using Medicare data, the researchers looked at how well hospitals did at keeping patients ages 65 to 99 from dying within 30 days of their operation, suffering a major complication that kept them in the hospital longer, having to have a repeated operation, or needing to be readmitted to the hospital again in their first month after going home. They also looked at the cost of patients' care, using a formula to account for general differences in hospital payments nationwide.

They looked at the three years before hospitals started participating in ACS-NSQIP, and at least two years following enrollment. Then, they compared the ACS-NSQIP hospitals' performance with that of non-ACS-NSQIP hospitals during the same period.

The authors also controlled for many other factors that could have influenced safety and outcomes -- other conditions the patients had, and factors about the hospital where they were treated, such as for-profit status, geographic region, bed size, teaching hospital status, and urban location.

No matter how they sliced the data, the result was the same: there was no improvement in any of four measures at ACS-NSQIP hospitals as compared to similar non-ACS-NSQIP hospitals. For instance, before ACS-NSQIP hospitals entered the program, 4.9 percent of their senior patients having these 11 operations died within 30 days of their operation, compared with 5 percent of those at non-participating hospitals. About one in 10 patients suffered a complication, about 13 percent went back to the hospital within 30 days, and 0.5 percent had to have a second operation. Rates were slightly higher at non-ACS-NSQIP hospitals analyzed in the study.

After three years of participation in ACS-NSQIP, the rates of all four measures had dropped at NSQIP hospitals -- but they had also dropped at the other hospitals. When the researchers adjusted improvement across all hospitals over time, there was no statistical difference between those treated at hospitals taking part in the ACS-NSQIP and those treated at comparison hospitals.

The cost of the patients' care, after adjustment, was also similar -- including payments for the initial hospital stay, and payments for additional stays and extraordinary "outlier" cases.