Bergstrom discusses negative results in scientific literature

"False facts" are being canonized in scientific literature due to the under-publication of negative results, said Carl Bergstrom, professor of biology at the University of Washington.

The lecture, titled “Modeling Scientific Activity,” consisted of a two-part discussion on scientific activity, the first being publication bias leading to the canonization of "false facts," and the second a game theoretic model of the questions that scientists choose to pursue.

Bergstrom explained that to accurately accept as true 99.9 percent of facts at a p-value of 0.05, over 40 percent of the obtained negative results need to be published. Negative results are those that show non-correlations between variables rather than correlations, or those that show non-differences between experiments rather than differences, he explained.

“If fewer than 10-40% of negative results are being published, we have a serious problem — yet reputed journals like ‘Cell,’ ‘Nature,’ and ‘Science’ never publish negative results,” Bergstrom said.

He added that editors of many scientific journals were dismissive of the fact that negative results could contain any information at all.

Bergstrom said that this means some reasonable fraction of the facts in our textbooks and review papers are false. Mathematical models show that the average reader is unable to gauge the truth of a claim if they see a disproportionate number of positive results, he explained.

Part of the problem, he said, was that scientists cannot build careers by publishing negative results.

“Perhaps we think that anyone can get a negative result, so there’s nothing great in that. No one ever received tenure after thirty experiments that don’t produce positive results,” Bergstrom said.

Such incentives as tenure and grants also influence the problems that institutions encourage scientists to work on, he added.

“Scientists these days do research to get grants rather than vice-versa,” Bergstrom said, adding that scientists today were epistemically sullied. “It’s no longer the picture of a lone scientist striving to understand the workings of the universe.”

Bergstrom explained that scientists working on a hot problem were in a “credit race” where the first person to solve the problem gets all the credit while the others get none at all. He used this as a basis to set up a game theory model of problems that scientists choose to work on, considering such factors as the scientific “capital” that a researcher has (for example, grants and previous experience), the value of the other problems that they are working on, and the payoff received for solving the problem.

Multiple Nash equilibria exist for such a problem, but only one minimizes the risk to any individual of dropping other research to pursue a hot topic, Bergstrom explained. The model could also change based on the publication of partial results, he added. When researchers publish results that partially solve the problem, they receive the credit for the part that they have solved, but then everyone else can catch up to the same point, after which the race is thrown wide open again.

Get the best of ‘the Prince’ delivered straight to your inbox. Subscribe now »

Bergstrom noted that analysis of when partial results were published and by whom shows that individuals with low scientific capital tend to publish immediately, while those with more experience and money tend to hold back and wait. If researchers were dissuaded from publishing partial results, many tended to opt out of solving the problem itself, he said.

The lecture was part of the EEB 522 Seminar Series: Colloquium on the Biology of Populations and took place in Guyot Hall on Thursday at 12:30.