UK report finds growing use of AI in evaluating university research quality

A new national report has shown for the first time how generative AI (GenAI) is already being used by some universities to assess the quality of their research—and it could be scaled up to help all higher education institutions (HEIs) save huge amounts of time and money. But the report, led by the University of Bristol, also reveals widespread skepticism among academics and professionals working in the sector about its usage for this purpose, and highlights the need for national oversight and governance.

The UK’s system for assessing the quality of research in higher education institutions (HEIs) is known as the Research Excellence Framework (REF). Its outcomes influence how around £2 billion annually of public funding is allocated for universities’ research. Lead author Richard Watermeyer, Professor of Higher Education at the University of Bristol, said, ”GenAI could be a game-changer for national-level research assessment, helping to create a more efficient and equitable playing field. Although there is a lot of vocal opposition to the incorporation of it into the REF, our report uncovers how GenAI tools are nevertheless being widely—if currently, quietly—used, and that expectation of their use by REF panelists is high.”

The last REF took place in 2021 and, following a review, changes to guidance for the next one—REF2029—are expected to be announced this month. Total costs of REF2021 are estimated to be around £471 million, with an average of £3 million per participating HEI, and REF2029 finances are anticipated to be much higher.
report has shown for the first time how generative AI (GenAI) is already being used by some universities to assess the quality of their research—and it could be scaled up to help all higher education institutions (HEIs) save huge amounts of time and money.

Prof Watermeyer said, ”The report is timely given the immense financial pressures facing the sector. It’s widely accepted that the regulatory burden of the REF is high and will, in all likelihood, only increase. Our report demonstrates that GenAI has the potential to alleviate some of this, but offers no complete solution. It could also create new bureaucratic challenges of its own, including establishing new requirements and protocols for its appropriate use.”

Key findings
The report investigated the usage of GenAI at 16 HEIs, including Russell Group universities and more recently established universities, across the UK. Findings indicated evidence of GenAI being widely deployed to prepare REF submissions in some capacity. But the extent and way it was being used greatly differed. For instance, some universities were using the tools to gather evidence of their research impact on the wider world and to help craft related stories. In others, there was evidence of in-house tools being developed to streamline REF processes or GenAI being used to review, assess, and score their research.

Prof Watermeyer added, ”There is clear variation, in fact, disparity in how HEIs can and might use these tools for competitive advantage in the REF. The extent to which institutions might profit from these tools is, as you might expect, linked to their level of resourcing and local expertise.” The study also included a survey of nearly 400 academics and professional services staff, which asked how they felt about GenAI tools being used for various aspects of REF2029.

In all respects, the majority of academics and professional services staff were shown to strongly disagree, with the level of strong opposition varying between 54% and 75% of respondents for different parts of the REF process. There was most support for GenAI tools being deployed, among almost a quarter (23%) of respondents, to support universities in the development of impact case studies.

Source: Phys.org