Journalists are accused of exaggerating, but new research finds they tend to temper, not exaggerate, scientific claims
The trope of the journalist writing in capital letters and distorting, exaggerating scientific findings is already well established. But while such examples exist (and they should be taxed), journalists in general are quite careful when reporting scientific findings – says exactly one scientific study.
In science, absolute certainty is often difficult to achieve. Uncertainty is a factor in the process and does not mean that a theory is wrong. Scientists have developed specialized, often obscure, methods for discussing scientific uncertainty, but outside the scientific community, these methods and associated terminology can be confusing and lead to erroneous conclusions.
This is especially true for science journalists, who must regularly report on scientific research and may sometimes be unaware of or misunderstand scientific uncertainty. However, this is the exception rather than the rule, according to a new study. The researchers found that journalists are generally cautious when reporting on science.
“I feel like when we talk about the potential for journalists exaggerating claims, it’s always these extreme cases,” David Jurgens, an assistant professor at the University of Michigan, said in a statement. “We wanted to see if there was a difference when we lined up what the scientist said and what the reporter said for the same article.”
Understanding science communication
Science journalists play an important role in shaping public understanding of science, as they broker information between the scientific community and the general public. They are intermediaries between scientists and the general public, and their judgment and expertise enable the public to engage with new scientific data. How they frame new scientific findings influences public opinion – because let’s face it, when was the last time you read an actual scientific study?
Many studies have looked at journalism in the face of scientific uncertainty over the years. Last year, a group of American researchers examined how journalists communicated preprints in the early moments of the Covid-19 pandemic, finding that half of the articles analyzed from digital media contained framing devices putting the emphasis on uncertainty.
In a new study, David Jurgens and Jiaxin Pai of the University of Michigan examined how scientific uncertainty is communicated by journalists and whether or not scientific claims are exaggerated. They also wanted to explore how scientific claims in the news have shifted between well-respected and less rigorous publications.
“Our findings suggest that journalists are actually quite careful when reporting science,” Pei said in a statement, highlighting the skills needed to translate science to a general audience. “Journalists have a tough job. It’s nice to see that they really try to contextualize and temper the scientific findings into a larger space.
The researchers gathered news data from Altmetrics, a company that tracks mentions of scientific articles in news stories. They collected about 129,000 articles mentioning scientific papers. In the articles, they analyzed sentences that included words such as “conclude” to see how the reporters expressed the newspaper’s claims.
They established levels of certainty in more than 1,500 scientific discoveries. Jurgens and Pai then built a computer model to see if they could replicate the levels of certainty reported by readers. The model was highly correlated with human ratings of the certainty of a claim. The model isn’t perfect, but it’s good enough to get an idea of what’s going on. Overall, the reporters stuck to the level of certainty presented in the study, the researchers note.
However, Pei said the job of reporters can sometimes get trickier when it comes to the quality of the paper. News writers tend to report the same levels of certainty no matter where the study was published, he argued. That can be problematic for the public, Pei said, because the journal’s impact factor is an indicator of research quality (although that’s also a matter of debate).
The researchers believe their work is a big step forward in understanding and quantifying how uncertainty is communicated in science news. To further help journalists and scientists, they have created software that helps calculate uncertainty in research and reporting. It could also benefit the general public, they claim, by providing a “calming effect” to some extent.
The study was published in the proceedings of the 2021 conference on empirical methods in natural language processing.