If research offices were still printing copies of REF guidance documents, yesterday would be a bumper day for the toner cartridges. The final Guidance on Submissions and Panel Criteria and Working Methods have been published by the REF team and come in at about 531 pages. The online format of the guidance may be less demanding on the printer, but it remains challenging to navigate.
Overall, the approach the guidance outlines is pragmatic, responsive to sector feedback and seems genuinely intended to reduce burden, although in practice any reduction may be relatively marginal in what remains an inherently burdensome process. On AI, there is a cautious approach to use in the assessment process and only for internal administrative processes: human judgement remains firmly in the foreground for assessing quality.
Final Guidance on Submissions: details, templates and flags
The main updates in the Guidance relate to the Contribution to Knowledge and Understanding (CKU) and – following the summer consultation process – Strategy, People and Research Environment (SPRE).
In the CKU section, much of the additional information works through the practicalities of implementing earlier decisions, particularly around eligibility and evidencing substantive links when individuals are decoupled from outputs. This includes information about how the HESA staff record will use REF fields for data collection. There is detail on the REF 2029 approach to data verification, which requires HEIs to be able to demonstrate outputs meet the eligibility criteria, and the reason for inclusion in a specific UoA. This may add to institutional burden, but should be aligned with processes already being undertaken in institutions, following their Codes of Practice. There is also a welcome addition of an option to use 300-word statements to describe the research and contributions of outputs where this is not evident from the submitted material itself.
In the Engagement and Impact section, the provision of a simplified template has raised some questions, particularly where the previous template had provided a useful practical tool to support preparation and drafting of an ICS. It is clear that the new shorter, more basic template is intended just for uploading into the submission system for REF 2029. In practice, institutions are likely to continue to use their existing templates for drafting case studies. Similarly, continuing case studies are welcomed and no longer need to be flagged separately – this should simplify submission of these case studies.
Following feedback from the sector in the summer consultation survey, the SPRE guidance has dropped the suggestion that institutions only submitting to a single UoA could submit a single institution-level template, with additional elements from the unit-level template. This section also includes links to institution-level (IL) and unit-level (UL) templates. However you need to look at appendices in the Panel Criteria and Working Methods for a clear view of which elements provide contextual information to inform assessment and which elements are directly scored as part of the assessment. Also featured for the first time are indicative lists of indicators, based on the learning from the People, Culture and Environment indicators project, such as qualitative/quantitative data evidencing effectiveness of strategic initiatives, or from surveys of staff and student experience. These can be used to evidence aspects of the IL and UL. Although it is made very clear that each section of “the narrative must be supported with indicators”, it is also clear that these lists of example indicators are non-exhaustive. Panels will also be provided with quantitative data on doctoral degrees awarded (2020-2028), research income (2020-2028), clinical academics (2020-2028). The problematic research income-in-kind dataset used in REF 2021 has been dropped.
Panel Criteria and Working Methods: a “one REF” approach
The Panel Criteria and Working Methods is a new publication. In particular, it emphasises what the REF team describes as the “one REF” approach to criteria and quality definitions – aiming for “a harmonised interpretation to reduce burden for HEIs”. There are still points of divergence in the Panel Criteria – for example citation data will be used differently in Panels A and B, only used in one sub-panel of Panel C and not used at all in Panel D – but it’s encouraging to see that there are fewer of these than was the case in REF 2021. Interdisciplinary outputs will be welcomed and no longer require a separate flag – sub-panels will use their judgement to identify interdisciplinary outputs for monitoring and analysis, and processes will ensure appropriate assessment of outputs crossing disciplinary boundaries, with support from interdisciplinary advisors.
What about the use of Artificial Intelligence?
The Guidance on Submissions is silent about use of AI by institutions in preparing their submissions, which is likely to lead to a wide range of different practices as institutions develop their own approaches. However, the Panel Criteria and Working Methods provide a clear and cautious statement about the Panels’ approach to the use of AI in their work, keeping the emphasis firmly on human judgement, with four protocols clearly outlined:
- confidentiality: no uploading of content to an AI tool.
- human judgement: no use of AI to make quality judgements “recognising that reliable quality assessment requires the exercise of human judgement”. Translations of outputs not in English will not be created using AI.
- data processing: there is reference to “automated processes” relating to tracking scoring patterns “with a view to improving the robustness of assessment” – and a guarantee that such use would be calibrated and “transparently communicated to the sector”.
- administrative efficiency: the REF team may use AI/automation tools for administrative processes such as allocating outputs to reviewers based on expertise matching. Again, this would be “transparently communicated” and not a substitute for human judgement.
The updated REF 2029 Timetable provides more detail for institutional planning between now and the submission deadline of 24 November 2028.
So on a first look, there are positive indications from both documents that sector feedback has been noted and that reducing the burden of REF remains a priority. There is also evidence that “fairness runs through the criteria more widely… designed to be sensitive to disciplinary and interdisciplinary context, so that excellence is recognised wherever it is found.”
