Research assessment reform efforts poised for breakthrough

It has been a decade since the San Francisco Declaration on Research Assessment (DORA) was declared to press for an overhaul in the ways in which outputs of scholarly research are evaluated.

In that time movements have sprung up around the world to campaign for a radical shake-up of a system that overvalues research outputs, particularly articles in prestigious academic journals and citations, rather than how research is done and how it benefits society.

Now, after months, if not years, of planning and detailed consultations with researchers and other stakeholders, a landmark in the campaign will be reached on Wednesday 28 September, 2022: the European University Association (EUA,) which speaks for more than 850 universities and rectors’ conferences in 49 countries; Science Europe, which represents major European public organisations funding ground-breaking research; and the European Commission will ask universities and other research organisations to sign an agreement pledging to reform the current research assessment system.

Organisations, which include public and private research funders, universities, research centres, institutes and infrastructures, associations and alliances, national and regional authorities, accreditation and evaluation agencies, learned societies and associations of researchers, and other relevant organisations, have been involved in drawing up the document.

The final version of the agreement was published in July and organisations involved in research assessment, based both in Europe and further afield, will be invited to add their signatures to the research assessment reform campaign document at the end of the first day of a research and innovation event hosted by the European Commission on 28 September.

A growing international movement

The development has been widely welcomed in research circles, including by DORA, which applauded the Europe-wide effort, saying they see “the agreement and coalition as an important part of the growing international movement for research assessment reform”.

However, a word of caution came from the League of European Research Universities (LERU).

Its spokesperson, Bert Overlaet, a professor at KU Leuven, Belgium, and chair of LERU’s careers policy group, told University World News that they while the league supported moves to reform research assessment in Europe and generally welcomed the resulting proposed agreement, the hard work has only just began.

“The European Commission decided not to include national governments in the earlier stages of these discussions, which may have been a wise decision to avoid endless delays, but we now need to get the member states and other governments, such as UK and Swiss, on board and work out how to fund the proposed changes,” he said.

“This requires more than just political lip service that change is necessary, but real engagement with the agreement and reform process.

“Some universities will be reluctant to sign the agreement or to join the coalition if they have no guarantee that their national government will provide support, both in funding and legislation.

“Changing the way researchers are assessed will require more resources from universities and funders than the present use of journal-based metrics. Universities therefore are right to ask how these efforts will be funded, as the funding of higher education and research has been going down in some countries in recent years.”

Inappropriate metrics

The agreement which universities and others will be asked to sign on 28 September doesn’t shy away from the challenges ahead and says a key goal is to “abandon inappropriate uses in research assessment of journal- and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index.”

It goes on to state: “This commitment will reduce the dominance of a narrow set of quantitative journal and publication-based metrics” and means moving away from using metrics like the JIF, Article Influence Score (AIS) and h-index as proxies for quality and impact.

The agreement states that “inappropriate uses” include relying exclusively on author-based metrics, such as counting papers, patents, citations and grants to assess quality and/or impact; assessing outputs based on metrics relating to publication venue, format or language; and relying on any other metrics that do not properly capture quality and/or impact.

It also wants to stop the use of rankings of research organisations in research assessment and seeks a commitment from signatories of the agreement to help avoid metrics being used by international rankings, which, it says, “are inappropriate for assessing researchers”.

It argues that the research community and research organisations need to “regain the autonomy to shape assessment practices, rather than having to abide by criteria and methodologies set by external commercial companies”.

The agreement says: “Research organisations should also be mindful that public communication (eg, the active advertising of an institution’s rank) can contribute to the perception that research quality conflates with ranking positions.”

It also asks research organisations and funders to commit to resourcing the reform process, “whether in the form of budget or staff capacity” and “to improve research assessment practices within [an] agreed timeframe”.

Good science practices

The European Commission is throwing its weight behind an awareness-raising campaign to win over universities and other stakeholders and is using social media tools, such as YouTube, to get across the message that reform is needed.

In one of its YouTube broadcasts, two up-and-coming young academic researchers were interviewed by Silvia Bottaro, a policy officer with the European Commission, about why reform is necessary.

Noémie Aubert Bonn, post-doctoral researcher at the University of Hasselt in Belgium and the University of Amsterdam medical centre, who was featured last year in a University World Newsstory on the need to shake up the reward and recognition of academic researchers, was one of those interviewed.

She said the very narrow set of metrics used at the moment forced researchers to choose between focusing on publications rather than their actual research and ignored other activities “that are extremely important in research, such as peer review, mentoring and supervision”.

She said changing the system would allow her and other researchers “to follow good science practices” and admitted there was a tension at the moment between “doing something to advance my career or doing something to advance science”.

“It is always very challenging, whereby if you do something to advance science you might not get the next grant,” she said.

Toma Susi, an associate professor at the University of Vienna in Austria, said in the video that reforming the system would be better for mental health and create more diversity. “We don’t simply have to publish or perish. This is a choice we can all make as a research community.”

A report in Science | Business on the European push to reform research assessment published on 22 September 2022, noted that “the European Commission got the ball rolling on the reform in the beginning of this year as part of moves to revive the European Research Area (ERA)” and that the process was endorsed by EU science ministers in June 2022.

Funding agencies, researchers, universities and organisations can start signing up to the agreement from 28 September to 15 November if they want to participate in the first big meeting of the coalition, scheduled to take place on 1 December 2022.

Quality and impact

Vinciane Gaillard, deputy director for research and innovation at the EUA, told University World News: “Recognising the quality and impact of research is at the core of our agreement, which looks beyond developments at a national level and provides a framework which will now be implemented through the coalition.”

She said the definitions of ‘impact’ in the European agreement and the UK’s REF 2021, which University World News reported on recently, are “very similar and very broad” and include “scientific, technological, economic, cultural and societal impact that may develop in the short-, medium- or long-term and cover both basic and frontier research and applied research.

“Research assessment by research funders should consider disciplinary, multi-, inter-, and trans-disciplinary research as well as contributions to knowledge generation and scientific, technological, economic, cultural and societal impact,” she said.

In a blog on the EUA website, Gaillard and her colleagues, Stephane Berghmans, director for R&I at the EUA, and research and innovation adviser, Rita Morais, discuss why universities should get involved with reforming research assessment.

Gaillard told University World News: “Reimagining research and academic assessment provides a unique opportunity to recognise a wide range of contributions. The academic sector now has a unique opportunity to reclaim ownership of research assessment and align it with core academic values such as academic freedom, institutional autonomy, research integrity, diversity and inclusion, cooperation, openness and knowledge sharing, critical thinking and democracy.”

Nic Mitchell is a UK-based freelance journalist and PR consultant specialising in European and international higher education. He blogs at www.delacourcommunications.com