Article (Archive)
James Heathers
In 2009, a now highly-cited study found an average of around 2% of scientists admit to have falsified, fabricated, or modified data at least once in their career.
Fifteen years on, a new analysis tried to quantify how much science is fake – but the real number may remain elusive, some observers said.
The analysis, published before peer review on the Open Science Framework on September 24, found one in seven scientific papers may be at least partly fake. The author, James Heathers, a long-standing scientific sleuth, arrived at that figure by averaging data from 12 existing studies — collectively containing a sample of around 75,000 studies — that estimate the volume of problematic scientific output.
“I have been reading for years and still continue to read this 2% figure which is ubiquitous,” Heathers, an affiliated researcher in psychology at Linnaeus University in Vaxjo, Sweden, said. “The only minor problem with it is that it’s 20 years out of date,” he added, noting that the last dataset that went into the 2009 study was from 2005.
So Heathers tried to come up with a more up-to-date estimate of scholarly literature containing signs of irregularities. “A lot has changed in 20 years,” he said. “It’s been a persistent irritant to me for a period of years now to see this figure cited over and over and over again.”
Past studies predominantly focussed on asking researchers directly if they had engaged in dishonest research practices, Heathers said, “which I think is a very bad approach to being able to do this.” But he noted that it was probably the only method available to use when the research was conducted.
“I think it’s pretty naive to ask people who are faking research whether or not they’ll honestly answer the question that they were dishonest previously,” Heathers said.
Heathers’ study pulls data from 12 different analyses from the social sciences, medicine, biology, and other fields of research. All those studies have one thing in common: The authors of each used various online tools to estimate the amount of fakery taking place in a set of papers.
“There’s a really persistent commonality to them,” Heathers said. “The rough approximation for where we end up is that one in seven research papers are fake.”
Heathers said he decided to conduct his study as a meta-analysis because his figures are “far flung.”
“They are a little bit from everywhere; it’s wildly nonsystematic as a piece of work,” he said.
Daniele Fanelli, a metascientist at Heriot-Watt University in Edinburgh, Scotland who authored the 2009 study, is not convinced by the new analysis. “Metascience research is sometimes not metascientific,” he says, arguing that the study falsely labels studies with some problem as definitely being a fake and incorrectly lumps together different studies measuring different phenomena.
“The papers are all different, they’re all over the place in highlighting all sorts of different problems in all sorts of different contexts using all sorts of different methods,” Fanelli says. “That’s not a rigorous way to get an estimate of anything.”
Fanelli said the study will draw unnecessary negative media attention: “It’s not the kind of attention that science either deserves or will benefit from.”
“I don’t think it’s entirely wrong but I think that it can be slightly misleading,” said Gowri Gopalakrishna, an epidemiologist at the Maastricht University in the Netherlands who co-authored a 2021 study that found 8% of researchers in a survey of nearly 7,000 scientists in the Netherlands confessed to falsifying or fabricating data at least once between 2017 and 2020.
Gopalakrishna said fabrication and falsification may be more prevalent in some fields than others so grouping them together may not be helpful. “If you want to get the attention of the government and try to shake things up, putting them all together and saying look how big the problem is probably useful in that way but I really do think that it’s important to drill down,” she said.
Heathers acknowledged those limitations but argued that he had to conduct the analysis with the data that exist. “If we waited for the resources necessary to be able to do really big systematic treatments of a problem like this within a specific area, I think we’d be waiting far too long,” he said. “This is crucially underfunded.”
Heathers said he decided to pursue coming up with a figure for the average percentage of fakery in science because few such estimates are available. “Even if you do something that’s an incredibly systematic review in a very formal sense, I strongly suspect you’ll get the same estimate that I’ve got,” he said.
James Heathers
In 2009, a now highly-cited study found an average of around 2% of scientists admit to have falsified, fabricated, or modified data at least once in their career.
Fifteen years on, a new analysis tried to quantify how much science is fake – but the real number may remain elusive, some observers said.
The analysis, published before peer review on the Open Science Framework on September 24, found one in seven scientific papers may be at least partly fake. The author, James Heathers, a long-standing scientific sleuth, arrived at that figure by averaging data from 12 existing studies — collectively containing a sample of around 75,000 studies — that estimate the volume of problematic scientific output.
“I have been reading for years and still continue to read this 2% figure which is ubiquitous,” Heathers, an affiliated researcher in psychology at Linnaeus University in Vaxjo, Sweden, said. “The only minor problem with it is that it’s 20 years out of date,” he added, noting that the last dataset that went into the 2009 study was from 2005.
So Heathers tried to come up with a more up-to-date estimate of scholarly literature containing signs of irregularities. “A lot has changed in 20 years,” he said. “It’s been a persistent irritant to me for a period of years now to see this figure cited over and over and over again.”
Past studies predominantly focussed on asking researchers directly if they had engaged in dishonest research practices, Heathers said, “which I think is a very bad approach to being able to do this.” But he noted that it was probably the only method available to use when the research was conducted.
“I think it’s pretty naive to ask people who are faking research whether or not they’ll honestly answer the question that they were dishonest previously,” Heathers said.
Heathers’ study pulls data from 12 different analyses from the social sciences, medicine, biology, and other fields of research. All those studies have one thing in common: The authors of each used various online tools to estimate the amount of fakery taking place in a set of papers.
“There’s a really persistent commonality to them,” Heathers said. “The rough approximation for where we end up is that one in seven research papers are fake.”
Heathers said he decided to conduct his study as a meta-analysis because his figures are “far flung.”
“They are a little bit from everywhere; it’s wildly nonsystematic as a piece of work,” he said.
Daniele Fanelli, a metascientist at Heriot-Watt University in Edinburgh, Scotland who authored the 2009 study, is not convinced by the new analysis. “Metascience research is sometimes not metascientific,” he says, arguing that the study falsely labels studies with some problem as definitely being a fake and incorrectly lumps together different studies measuring different phenomena.
“The papers are all different, they’re all over the place in highlighting all sorts of different problems in all sorts of different contexts using all sorts of different methods,” Fanelli says. “That’s not a rigorous way to get an estimate of anything.”
Fanelli said the study will draw unnecessary negative media attention: “It’s not the kind of attention that science either deserves or will benefit from.”
“I don’t think it’s entirely wrong but I think that it can be slightly misleading,” said Gowri Gopalakrishna, an epidemiologist at the Maastricht University in the Netherlands who co-authored a 2021 study that found 8% of researchers in a survey of nearly 7,000 scientists in the Netherlands confessed to falsifying or fabricating data at least once between 2017 and 2020.
Gopalakrishna said fabrication and falsification may be more prevalent in some fields than others so grouping them together may not be helpful. “If you want to get the attention of the government and try to shake things up, putting them all together and saying look how big the problem is probably useful in that way but I really do think that it’s important to drill down,” she said.
Heathers acknowledged those limitations but argued that he had to conduct the analysis with the data that exist. “If we waited for the resources necessary to be able to do really big systematic treatments of a problem like this within a specific area, I think we’d be waiting far too long,” he said. “This is crucially underfunded.”
Heathers said he decided to pursue coming up with a figure for the average percentage of fakery in science because few such estimates are available. “Even if you do something that’s an incredibly systematic review in a very formal sense, I strongly suspect you’ll get the same estimate that I’ve got,” he said.
