{"id":9266,"date":"2021-07-13T20:52:22","date_gmt":"2021-07-13T20:52:22","guid":{"rendered":"https:\/\/serrapilheira.org\/the-naked-truth-about-peer-review\/"},"modified":"2021-07-13T20:54:55","modified_gmt":"2021-07-13T20:54:55","slug":"the-naked-truth-about-peer-review","status":"publish","type":"post","link":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/","title":{"rendered":"The naked truth about peer review"},"content":{"rendered":"

Academic science\u2019s quality filter emanates authority but means little<\/span><\/i><\/p>\n

\"\"<\/span><\/p>\n

By Olavo Amaral<\/strong><\/p>\n

The original article was published on Folha de S.Paulo<\/a>.<\/em><\/p>\n

You can bet that in any discussion of scientific data, sooner or later someone will invoke the \u201cpeer-reviewed article\u201d argument, either to give credence to a statement or to discredit it, if such a review did not take place.<\/span><\/p>\n

Peer review\u2014approval by independent researchers before the publication of an article\u2014has been considered a bastion of scientific research for decades (<\/span>or more than a century, depending on the field<\/span><\/a>), and for many, it defines what it is considered \u201cscience\u201d and what is not.<\/span><\/p>\n

In an <\/span>iconic image<\/span><\/a> from the March for Science in Washington in 2017, a sign reading \u201cIn peer review we trust\u201d, an allusion to \u201cIn God we<\/span> trust\u201d, was seen in front of the Capitol. The substitution, however, is equivalent to exchanging one dogmatic belief for another.<\/span><\/p>\n

\u201cPeer-reviewed\u201d, after all, only means that some people\u2014usually two or three\u2014analyzed an article and <\/span>saw no reason to deny its publication<\/span><\/a>. As the process usually occurs <\/span>behind closed doors<\/span><\/a>, we do not know who these people are, what opinions they expressed, or what they bothered to verify.<\/span><\/p>\n

Aside from this, reviewers are not usually <\/span>trained for the task<\/span><\/a>, <\/span>are given no direction on what to review<\/span><\/a> and are <\/span>not paid or rewarded for their work<\/span><\/a>, thus having little support or encouragement as they undertake a review. It is not surprising that the agreement between different reviewers is minimal and sometimes <\/span>borders on random<\/span><\/a>.<\/span><\/p>\n

As if that were not enough, reviewers act only <\/span>at the end of the scientific process<\/span><\/a>, when problems in data collection are impossible to fix. Worse still, reviewers work based on the authors\u2019 report and generally do not have access to the original data, which prevents them from detecting most errors and omissions that may occur throughout a study.<\/span><\/p>\n

If none of this makes you suspect that something is wrong, imagine applying the same logic in other areas. If an airline told you that it delegates its quality control to two or three experts who examine a few pages of a report of on the manufacturing of a plane that has already been built, would you board it?<\/span><\/p>\n

The <\/span>confidence of the scientific community in peer review<\/span><\/a> is even more disconcerting given the scarce evidence of its impact on the scientific literature. Comparisons between preprints\u2014articles posted before peer review\u2014and their reviewed versions show that the <\/span>differences in quality are small<\/span><\/a> and that both the <\/span>text<\/span><\/a> and the <\/span>main conclusions<\/span><\/a> rarely change.<\/span><\/p>\n

Regarding the filtering function, the failure of the system is even more striking. <\/span>Nonsensical<\/span><\/a> articles, with <\/span>gross errors<\/span><\/a> or <\/span>preposterous conclusions<\/span><\/a>, written as jokes, invariably <\/span>end up being accepted somewhere<\/span><\/a>. The problem is aggravated by the so-called \u201c<\/span>predatory journals<\/span><\/a>\u201d\u2014journals that charge per publication and maximize their profits through a lack of rigor.<\/span><\/p>\n

The COVID-19 pandemic is replete with examples of the weakness of the system. Supposedly peer-reviewed journals published far-fetched theories such as <\/span>5G technology being able to produce SARS-CoV-2<\/span><\/a>. Meanwhile, journals with editors affiliated with Didier Raoult\u2019 Institut Hospitalo-Universitaire M\u00e9diterran\u00e9e Infection have become a <\/span>biased showcase of studies advocating the use of hydroxychloroquine<\/span><\/a>.<\/span><\/p>\n

It would be easy to attribute the problem to low-quality journals, but the most notorious pandemic scandal hit <\/span>The Lancet<\/span><\/i><\/a> and the <\/span>New England Journal of Medicine<\/span><\/i><\/a>, the most respected medical journals in the world, which were forced to retract articles with <\/span>data suspected of being fabricated<\/span><\/a> by the company Surgisphere.\u00a0<\/span><\/p>\n

This is not surprising: although traditional journals tend to be more selective in accepting articles, there is nothing different about their review processes. In addition, the <\/span>pressure to publish in these journals<\/span><\/a> may encourage scientists to sugarcoat data to make their results more attractive. Thus, using <\/span>“impact factor” as a quality criterion<\/span><\/a> does not solve the problem: visibility and reliability are different things, after all.<\/span><\/p>\n

In the Surgisphere situation, critics were quick to point out culprits, such as <\/span>editors\u2019 bias<\/span><\/a> and <\/span>reviewers\u2019 haste<\/span><\/a>. In truth, however, the review system itself is responsible, and that system, without access to the data or the process by which they were obtained, <\/span>does not have the ability to identify well-devised frauds<\/span><\/a>.<\/span><\/p>\n

If peer review does not serve as a yardstick, what can be considered \u201cscientifically supported\u201d? The best answer, somewhat tautological, may be \u201cscientific consensus\u201d. However, identifying this is not always obvious. The positions of scientific institutions and societies are an approximation of this, but they have their political side\u2014which in cases such as that of Brazilian medical associations usually <\/span>flirt with syndicalism<\/span><\/a>\u2014and are far from bias-free.<\/span><\/p>\n

The truth is that we do not have efficient institutional ways of defining what reliable science is, representing a huge gap in public debate. This is evident in the headache of <\/span>fact-checking agencies<\/span><\/a> that deal with the dozens of articles for and against early treatment for COVID-19, a question too complicated to be summarized as <\/span>\u201c#fact\u201d or \u201c#fake\u201d<\/span><\/a>.<\/span><\/p>\n

There is much to do, therefore, to develop a hallmark that goes beyond the \u201cpeer-reviewed\u201d seal. This will only be achieved if we overcome the belief that two or three reviewers examining a PDF is enough to assess the quality of a complex process such as scientific research.<\/span><\/p>\n

Examples of success abound: audits, certifications and standard procedures are part of the routine of airports, hospitals and public infrastructure, and it is unclear why they are so rare in academic institutions. Even projects such as <\/span>Wikipedia<\/span><\/a> have more elaborate and robust review and correction processes than the anemic and opaque peer review of scientific articles.<\/span><\/p>\n

Without better controls, academic research will remain vulnerable to fraud, errors and biases, fueling quackery with the \u201cscientifically supported\u201d seal. This is simply the natural consequence of believing in a process in which no one sees what is being done. As in the <\/span>children’s story<\/span><\/a>, the emperor is naked under the invisible clothes of peer review, and sometimes it takes a child, or a pandemic, to force us to admit it.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Academic science\u2019s quality filter emanates authority but means little By Olavo Amaral The original article was published on Folha de S.Paulo. You can bet that in [\u2026]<\/span><\/p>\n","protected":false},"author":6,"featured_media":9143,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1184],"tags":[1712,1256,1710,1702],"class_list":["post-9266","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-fundamental-science","tag-peer-review-en","tag-science-en","tag-scientific-data","tag-scientific-publications"],"acf":[],"yoast_head":"\nThe naked truth about peer review - Instituto Serrapilheira<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The naked truth about peer review - Instituto Serrapilheira\" \/>\n<meta property=\"og:description\" content=\"Academic science\u2019s quality filter emanates authority but means little By Olavo Amaral The original article was published on Folha de S.Paulo. You can bet that in [\u2026]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/\" \/>\n<meta property=\"og:site_name\" content=\"Instituto Serrapilheira\" \/>\n<meta property=\"article:published_time\" content=\"2021-07-13T20:52:22+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-07-13T20:54:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/serrapilheira.org\/wp-content\/uploads\/2021\/06\/corte.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2663\" \/>\n\t<meta property=\"og:image:height\" content=\"1930\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"pedrolira\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"pedrolira\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/\"},\"author\":{\"name\":\"pedrolira\",\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/#\\\/schema\\\/person\\\/d3ca326d195471d40fa9401d9de937dd\"},\"headline\":\"The naked truth about peer review\",\"datePublished\":\"2021-07-13T20:52:22+00:00\",\"dateModified\":\"2021-07-13T20:54:55+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/\"},\"wordCount\":1014,\"image\":{\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/serrapilheira.org\\\/wp-content\\\/uploads\\\/2021\\\/06\\\/corte.png\",\"keywords\":[\"peer review\",\"science\",\"scientific data\",\"Scientific publications\"],\"articleSection\":[\"Column Fundamental Science\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/\",\"url\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/\",\"name\":\"The naked truth about peer review - Instituto Serrapilheira\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/serrapilheira.org\\\/wp-content\\\/uploads\\\/2021\\\/06\\\/corte.png\",\"datePublished\":\"2021-07-13T20:52:22+00:00\",\"dateModified\":\"2021-07-13T20:54:55+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/#\\\/schema\\\/person\\\/d3ca326d195471d40fa9401d9de937dd\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/#primaryimage\",\"url\":\"https:\\\/\\\/serrapilheira.org\\\/wp-content\\\/uploads\\\/2021\\\/06\\\/corte.png\",\"contentUrl\":\"https:\\\/\\\/serrapilheira.org\\\/wp-content\\\/uploads\\\/2021\\\/06\\\/corte.png\",\"width\":2663,\"height\":1930},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/the-naked-truth-about-peer-review\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"In\u00edcio\",\"item\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The naked truth about peer review\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/\",\"name\":\"Instituto Serrapilheira\",\"description\":\"Instituto de apoio \u00e0 ci\u00eancia no Brasil\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/#\\\/schema\\\/person\\\/d3ca326d195471d40fa9401d9de937dd\",\"name\":\"pedrolira\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/7fdf0800ae5535b41abd7ba8b296eb9c54ca21617e9500a035094eeb87225682?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/7fdf0800ae5535b41abd7ba8b296eb9c54ca21617e9500a035094eeb87225682?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/7fdf0800ae5535b41abd7ba8b296eb9c54ca21617e9500a035094eeb87225682?s=96&d=mm&r=g\",\"caption\":\"pedrolira\"},\"url\":\"https:\\\/\\\/serrapilheira.org\\\/en\\\/author\\\/pedrolira\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The naked truth about peer review - Instituto Serrapilheira","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/","og_locale":"en_US","og_type":"article","og_title":"The naked truth about peer review - Instituto Serrapilheira","og_description":"Academic science\u2019s quality filter emanates authority but means little By Olavo Amaral The original article was published on Folha de S.Paulo. You can bet that in [\u2026]","og_url":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/","og_site_name":"Instituto Serrapilheira","article_published_time":"2021-07-13T20:52:22+00:00","article_modified_time":"2021-07-13T20:54:55+00:00","og_image":[{"width":2663,"height":1930,"url":"https:\/\/serrapilheira.org\/wp-content\/uploads\/2021\/06\/corte.png","type":"image\/png"}],"author":"pedrolira","twitter_card":"summary_large_image","twitter_misc":{"Written by":"pedrolira","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/#article","isPartOf":{"@id":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/"},"author":{"name":"pedrolira","@id":"https:\/\/serrapilheira.org\/en\/#\/schema\/person\/d3ca326d195471d40fa9401d9de937dd"},"headline":"The naked truth about peer review","datePublished":"2021-07-13T20:52:22+00:00","dateModified":"2021-07-13T20:54:55+00:00","mainEntityOfPage":{"@id":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/"},"wordCount":1014,"image":{"@id":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/#primaryimage"},"thumbnailUrl":"https:\/\/serrapilheira.org\/wp-content\/uploads\/2021\/06\/corte.png","keywords":["peer review","science","scientific data","Scientific publications"],"articleSection":["Column Fundamental Science"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/","url":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/","name":"The naked truth about peer review - Instituto Serrapilheira","isPartOf":{"@id":"https:\/\/serrapilheira.org\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/#primaryimage"},"image":{"@id":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/#primaryimage"},"thumbnailUrl":"https:\/\/serrapilheira.org\/wp-content\/uploads\/2021\/06\/corte.png","datePublished":"2021-07-13T20:52:22+00:00","dateModified":"2021-07-13T20:54:55+00:00","author":{"@id":"https:\/\/serrapilheira.org\/en\/#\/schema\/person\/d3ca326d195471d40fa9401d9de937dd"},"breadcrumb":{"@id":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/#primaryimage","url":"https:\/\/serrapilheira.org\/wp-content\/uploads\/2021\/06\/corte.png","contentUrl":"https:\/\/serrapilheira.org\/wp-content\/uploads\/2021\/06\/corte.png","width":2663,"height":1930},{"@type":"BreadcrumbList","@id":"https:\/\/serrapilheira.org\/en\/the-naked-truth-about-peer-review\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"In\u00edcio","item":"https:\/\/serrapilheira.org\/en\/"},{"@type":"ListItem","position":2,"name":"The naked truth about peer review"}]},{"@type":"WebSite","@id":"https:\/\/serrapilheira.org\/en\/#website","url":"https:\/\/serrapilheira.org\/en\/","name":"Instituto Serrapilheira","description":"Instituto de apoio \u00e0 ci\u00eancia no Brasil","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/serrapilheira.org\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/serrapilheira.org\/en\/#\/schema\/person\/d3ca326d195471d40fa9401d9de937dd","name":"pedrolira","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/7fdf0800ae5535b41abd7ba8b296eb9c54ca21617e9500a035094eeb87225682?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/7fdf0800ae5535b41abd7ba8b296eb9c54ca21617e9500a035094eeb87225682?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/7fdf0800ae5535b41abd7ba8b296eb9c54ca21617e9500a035094eeb87225682?s=96&d=mm&r=g","caption":"pedrolira"},"url":"https:\/\/serrapilheira.org\/en\/author\/pedrolira\/"}]}},"_links":{"self":[{"href":"https:\/\/serrapilheira.org\/en\/wp-json\/wp\/v2\/posts\/9266","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/serrapilheira.org\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/serrapilheira.org\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/serrapilheira.org\/en\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/serrapilheira.org\/en\/wp-json\/wp\/v2\/comments?post=9266"}],"version-history":[{"count":0,"href":"https:\/\/serrapilheira.org\/en\/wp-json\/wp\/v2\/posts\/9266\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/serrapilheira.org\/en\/wp-json\/wp\/v2\/media\/9143"}],"wp:attachment":[{"href":"https:\/\/serrapilheira.org\/en\/wp-json\/wp\/v2\/media?parent=9266"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/serrapilheira.org\/en\/wp-json\/wp\/v2\/categories?post=9266"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/serrapilheira.org\/en\/wp-json\/wp\/v2\/tags?post=9266"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}