An experiment puts auditing under scrutiny

The structure of the auditing business appears problematic: Typically, major companies pay auditors to examine their books under the so-called "third-party" audit system. But when an auditing firm's revenues come directly from its clients, the auditors have an incentive not to deliver bad news to them.

So: Does this arrangement affect the actual performance of auditors?

In an eye-opening experiment involving roughly 500 industrial plants in the state of Gujarat, in western India, changing the auditing system has indeed produced dramatically different outcomes -- reducing pollution, and more generally calling into question the whole practice of letting firms pay the auditors who scrutinize them.

"There is a fundamental conflict of interest in the way auditing markets are set up around the world," says MIT economist Michael Greenstone, one of the co-authors of the study, whose findings are published today in the Quarterly Journal of Economics. "We suggested some reforms to remove the conflict of interest, officials in Gujarat implemented them, and it produced notable results."

The two-year experiment was conducted by MIT and Harvard University researchers along with the Gujarat Pollution Control Board (GPCB). It found that randomly assigning auditors to plants, paying auditors from central funds, double-checking their work, and rewarding the auditors for accuracy had large effects. Among other things, the project revealed that 59 percent of the plants were actually violating India's laws on particulate emissions, but only 7 percent of the plants were cited for this offense when standard audits were used.

Across all types of pollutants, 29 percent of audits, using the standard practice, wrongly reported that emissions were below legal levels.

The study also produced real-world effects: The state used the information to enforce its pollution laws, and within six months, air and water pollution from the plants receiving the new form of audit were significantly lower than at plants assessed using the traditional method.

The co-authors of the paper are Greenstone, the 3M Professor of Environmental Economics at MIT; Esther Duflo, the Abdul Latif Jameel Professor of Poverty Alleviation and Development Economics at MIT; Rohini Pande, a professor of public policy at the Harvard Kennedy School; and Nicholas Ryan PhD '12, now a visiting postdoc at Harvard.

The power of random assignment

The experiment involved 473 industrial plants in two parts of Gujarat, which has a large manufacturing industry. Since 1996 the GPCB has used the third-party audit system, in which auditors check air and water pollution levels three times annually, then submit a yearly report to the GPCB.

To conduct the study, 233 of the plants tried a new arrangement: Instead of auditors being hired by the companies running the power plants, the GPCB randomly assigned them to plants in this group. The auditors were paid fixed fees from a pool of money; 20 percent of their audits were randomly chosen for re-examination. Finally, the auditors received incentive payments for accurate reports.

In comparing the 233 plants using the new method with the 240 using the standard practice, the researchers uncovered that almost 75 percent of traditional audits reported particulate-matter emissions just below the legal limit; using the randomized method, only 19 percent of plants fell in that narrow band.

All told, across several different air- and water-pollution measures, inaccurate reports of plants complying with the law dropped by about 80 percent when the randomized method was employed.

The researchers emphasize that the experiment enabled the real-world follow-up to occur.

"The ultimate hope with the experiment was definitely to see pollution at the firm level drop," Duflo says. The state's enforcement was effective, as Pande explains, partly because "it becomes cheaper for some of the more egregious pollution violators to reduce pollution levels than to attempt to persuade auditors to falsify reports."

According to Ryan, the Gujarat case also dispels myths about the difficulty of enforcing laws, since the experiment "shows the government has credibility and will."

But how general is the finding?

In the paper, the authors broaden their critique of the audit system, referring to standard corporate financial reports and the global debt-rating system as other areas where auditors have skewed auditing incentives. Still, it is an open question how broadly the current study's findings can be generalized.

"It would be a mistake to assume that quarterly financial reports for public companies in the U.S. are exactly the same as pollution reports in Gujarat, India," Greenstone acknowledges. "But one thing I do know is that these markets were all set up with an obvious fundamental flaw -- they all have the feature that the auditors are paid by the firms who have a stake in the outcome of the audit."

To be sure, many large corporations have complicated operations that cannot be audited in the manner of emissions; in those cases, a counterargument goes, retaining the same auditor who knows the firm well may be a better practice. But Metrick suggests that in such cases, auditors could be randomly assigned to firms for, say, five-year periods. At a minimum, he notes, the Dodd-Frank law on financial regulation mandates further study of these issues.

Greenstone also says he hopes the current finding will spur related experiments, and gain notice among regulators and policymakers.

"No one has really had the political will to do something about this," Greenstone says. "Now we have some evidence."

The study was funded by the Center for Energy and Environmental Policy Research, the Harvard Environmental Economics Program, the International Growth Centre, the International Initiative for Impact Evaluation, the National Science Foundation and the Sustainability Science Program at Harvard.