In May, 2013, PLOS became one of the original 75 organizations to sign the Declaration of Research Assessment (DORA), an initiative that seeks to improve the way we assess the quality and impact of scientific research. We are a proud signatory because it was just four years ago that PLOS placed comprehensive performance indicators, called Article-Level Metrics (ALMs), on every article we publish, to track how it is being read, discussed, and cited, making it easier for the community to evaluate its importance. Today, many publishers are adopting this approach, and new businesses have been formed, further enriching the landscape of research impact assessment.
For many years, the impact of research articles has been assumed to be a simple reflection of the prestige of the journal in which an article is published. But many studies and numerous commentators have shown that this is not a reliable way to measure individual research articles, and is even less useful as a way of assessing the output of individual scientists. Every metric, including citations, has its pros and cons, but taken together a combination of metrics can give a more comprehensive evaluation of the true impact of an article.
In the past, the only way we knew if articles were being read was through the paper trail of those who had checked it out from the library or circulation list, and in due course if the article was cited. Today when people discover, read, and discuss papers they leave behind digital footprints, which include usage, citations and social sharing, so it makes sense to evaluate research via this broader range of information using ALMs. The more comprehensive scope of ALMs has other distinct advantages over traditional measures such as the transparency of data sources; the fact that all articles qualify for them upon publication, and they can be adopted by others through an API where available.
Encouraging the entire community to think about how best to integrate ALMs more fully into research evaluation is a key goal for PLOS:
Researchers – provide a reliable way to measure the real-time influence and reach of articles that can be set in the context of other content in similar fields of the same age. This information can be featured on a resume and shared with collaborators, institutions and funders.
Publishers – add value to a broad range of audiences including authors, librarians and institutions, by providing intelligence on the impact of published research. Publishers can also track the downstream distribution of content and use that knowledge to make better business decisions.
Institutions – offer a comprehensive view of an individual’s past and current work and reflect up-to-date measures of impact and output, helping institutions make solid hiring, tenure and promotion decisions. PLOS has developed a reporting tool to mine PLOS ALM data that provides organizations with customized reports and visualizations for download.
Funders – obtain business intelligence to provide evidence on how investments in people or projects are producing wider engagement or high-impact outputs. Funders can use the same reporting tool to aggregate and analyze metrics in ways that suit their funding criteria. Those who are grant recipients can also easily communicate and report their research progress with solid and measurable data.
To develop wider adoption of Article Level Metrics and best practices for their application, PLOS believes it is crucial that quantitative measures of article usage be provided in a consistent, coherent, and comprehensive manner. We believe that this will be best achieved through the creation of a shared community that aggregates and validates metrics from publishers and others and provides a trusted community resource for all stakeholders. Such a clearinghouse for ALM data will need community support and the adoption of a set of shared principles on the generation, validation, and use of the data.
As part of the PLOS mission to lead a transformation in research communication, we’re committed to the continued development of ALMs with publishers, funders, and research institutions to make them available and useful for all. PLOS continues to highlight the value of ALMs. We released the first iteration of Relative Metrics last month to provide much-needed context to our existing suite of ALMs. Now the community can compare average usage for similar articles published in the same time period. We have also added a new tool to our ALM repertoire called ALM Reports, which enables users to build sets of articles that interest them and download data reports and charts for further analysis.
To sum up, our view is that articles should be judged on their own merits, not on the journal in which they were published. Ultimately, such an approach will lead to a more accurate and timely assessment of research by providing real-time tracking of an article’s reach and performance, from the earliest downloads immediately after publication to the later stages of citations and article usage over time.