METRICS: A Little Backdrop
In 2005, the Public Library of Science (PLoS) Medicine journal published a paper by Greek researcher Dr. John Ioannidis entitled “Why most published research findings are false.” The paper went on to become the most downloaded article in PLoS history, with over 1 million views to date. It addressed research concerns that have dominated Ioannidis’ work for most of his career—poorly designed studies with questionable statistical inferences from small population samples, and the inability to reproduce study results across multiple disciplines.
In April 2014, Stanford University announced that Dr. Ioannidis’ work would have a new home as he agreed to become co-director with Dr. Steven Goodman of the new Meta-Research Innovation Center at Stanford (METRICS).
Funded with a $6 million grant from the Laura and John Arnold Foundation (LJAF), the METRICS center “aims to transform research practices to improve the reproducibility, efficiency, and quality of scientific investigations.”
Waging War on Sloppy Science
Questionable methodologies notwithstanding, the goal of improved efficiencies has the potential to impact research funding significantly. A 2014 Economist article quoted the medical journal Lancet as estimating that “In 2010, about $200 billion (an astonishing 85% of the world’s spending on medical research) was squandered on studies that were flawed in design, redundant, never published or poorly reported.” Improved research practices can certainly help to address such poor fiscal stewardship but attending to issues with redundancy and publishing may be beyond the purview of a medical research center.
Meta-Research: Research on Research
By definition, meta-research analysis will examine multiple studies to identify commonalities that will become the top priorities of the METRICS center. The interest in reproducibility of research drew the attention of the LJAF, and concerns over the perceived publication bias of journals skewing articles towards more counterintuitive studies and results remains a major concern for METRICS. The leadership of both Ioannidis and Goodman in this area may prove to be something of a mixed blessing. Their ideas will no doubt be well received by scientific colleagues, but policymakers and publishers may prove somewhat less receptive to dramatic change. In addition, METRICS itself will be under scrutiny to ensure that all research studies practice what they preach in terms of recommended methodologies.
A Return to Reliability
There appear to be very high expectations of the work of this center. Concerns within the academic community about journal retractions, questionable peer review practices, open access models, and other examples of scientific misconduct have created a general malaise about the perceived reliability of scientific research at a time when competitive funding bids cannot afford the risk of being overshadowed by questions of integrity. Since no generally accepted solutions have been presented yet, there is clearly room for discussion of both improved methodologies and models but much of that work remains to be seen.
