New, tougher standards for publishing scientific results

Last month, the Bush Administration released a new set of standards on the publication of scientific research by federal agencies. These new regulations, issued in compliance with the Treasury and General Government Appropriations Act for Fiscal Year 2001, tighten standards for publishing scientific results by federal agencies. They require agencies to ensure that all results published are "capable of being substantially reproduced."

To rule out lab error and faulty analyses, scientists usually attempt to reproduce experimental results published by their peers. Hence, the burden of ensuring that all good science is capable of being reproduced has typically fallen on the entire scientific community.

The new regulations are somewhat controversial because they place more of a burden on federal agencies to ensure the validity of their own data.

The new guidelines, which are to be implemented by federal agencies no later than one year after their issuance, seek to ensure and maximize "the quality, objectivity, utility, and integrity of information (including statistical information) disseminated" by federal agencies.

These agencies are also required to establish administrative mechanisms to allow individuals to seek and obtain the correction of any published research that is not in compliance with the new guidelines.

The University "certainly agree[s] with the principle of the new legislation, and support the conduct of research at the highest standards, with sound and informed decision making," said Michelle Christy, director of the Office of Research and Projects Administration, which oversees grant and contracts for research and projects funded by external sponsors.

The University spent slightly more than $133 million in research expenditures in the last academic year, which ran from July 1, 2000 to June 30, 2001. Roughly $85 million came from approximately 15 different federal research sponsors and other government sources, Christy said.

Physics department chair Dan Marlow said that while the impact on the University will be "pretty minimal" because the new guidelines do not impact federal funding, there is concern in universities about the new regulations.

"It always makes us nervous when the government gets in the business of regulating how research is to be done," he said. "The scientific community has a long established system of peer review that seems to work pretty well."

"If the government really got into the act of validating research, I personally fear that what for the most part is now a common sense approach to scientific review would be replaced by in inflexible system that would ultimately consume a lot of scientist's time without really raising the quality of research," Marlow added.

Christy said that since the federal agencies have up to a year to craft new guidelines to maintain compliance with the new regulations, the University does not yet know what each agency's policies will look like.

Get the best of the ‘Prince’ delivered straight to your inbox. Subscribe now »

In response to comments to early drafts of the new regulations, the Office of Management and Budget, which issued the regulations, gave leeway to federal agencies to determine the "appropriate level of correction for a complaint received" about the validity of published scientific results.

Through the University's participation on the Council on Governmental Relations, it helps to form policies with government agencies, and it will use its position on the board to continue to work to assure that the University can comply with these new guidelines as implemented by each federal agency, Christy added.

Among the University's goals are to ensure that whatever policies are designed are consistent with existing standards such as the Paperwork Reduction Act and don't incur significant added costs as a result of these changes.

Physics professor William Happer, former director of the office of energy research in the Department of Energy, said that the new guidelines would not have much impact on research in general.

"Having been a federal bureaucrat myself, I doubt that the new standard will make much difference to what is published, although there may be a few exceptions. The concern the standards are addressing is studies for which the raw data has been hoarded so that no one can check how well the statistical analysis has been done, what data might have been thrown out, perhaps to support some preconceptions about what the answer might be," he said.

Happer oversaw a three billion dollar research budget when he worked in the government from 1991 to 1993.

Marlow was more concerned that a system that formally punishes scientists for honest mistakes despite taking reasonable precautions would discourage innovation, but he said there appeared to be little to worry about over these particular regulations.

"The regulation in question does not look like it would do that, since the main remedy it mentions is asking that the research be repeated æ usually done for any new and surprising result anyway," Marlow said.

"Once again, however, one worries about the precedent, which is why universities are likely to kick up a little bit of a fuss about this." he explained. "The purpose of this regulation is to prevent government agencies from promulgating regulations that are not justified by scientific evidence — I am pretty sure that things like global warming are what they have in mind."

Happer agreed that there may have been abuses in some environmental studies and said that a potentially more serious problem has been clinical trials supported by pharmaceutical houses, "where some research results have been held back from publication if they cast doubt on the efficacy of certain products."

Otherwise, federally-funded research has been pretty honest and capable of being checked most of the time, Happer added.

Marlow said, "Put another way, I don't expect to have federal marshalls knocking down my door in the middle of the night because someone found an algebra mistake in one of our papers.

"I also note that every good scientist aspires to publish papers that are correct and contain reproducible results," he added. "In that sense, the government is not asking anyone to do anything that they shouldn't be doing already."