A conversation with Ian Carter and Pauline Muya (Part 1)
With the next Research Excellence Framework (REF 2029) on the horizon and institutions grappling with uncertainty and financial pressures, we spoke to two of the UK’s most experienced REF practitioners about the current landscape and what institutions should be prioritising. In this first of two blog posts, Ian Carter (Carter Research Navigation) and Pauline Muya (Muya Research Management) share their insights on the evolving REF process and current sector challenges.
Rob Johnson: Could you both briefly introduce yourselves and share your experience with REF assessments?
Pauline Muya: I’m Pauline Muya. I’ve been in research management for the last 32 years and my experience with REF specifically goes back to the 1996 exercise, and each exercise from then onwards. I’ve worked in half a dozen different universities in substantive roles in research management, most of which have had responsibility for REF preparations.
The two most recent were at the Institute of Education for the 2014 exercise where I was the professional services lead managing the whole REF submission, and likewise at the University of Leicester for the 2021 exercise from 2016 to 2019. At Leicester, there was a real pressure to perform and increase their ranking positions for the 2021 exercise. I’ve been freelance for the last five years supporting clients with REF and also worked as a panel advisor with Research England for the 2021 exercise.
Ian Carter: I’m Ian Carter. I’m trying to be retired and not necessarily being completely successful. Similar to Pauline, I’ve had an executive career and then moved into being an independent consultant. I’ve worked for three universities and then advised a range of institutions on research assessment, both RAE [Research Assessment Exercise, the precursor to REF] and REF.
I’ve been involved since 1996, so I’ve had five exercises. Four of those were as the institutional lead for the submission. For REF 2021, I advised about seven or eight different institutions on parts of their submission. I’ve also been involved in all of the consultation processes and some informal discussions that have potentially influenced parts of the exercises.
Rob Johnson: You’re both drawing on almost 30 years’ experience and five RAE/REF cycles. Looking at that holistically, what have been the most significant changes over that period?
Ian Carter: I think there are two different types of change. First, the REF in 2014 transitioned away from the old RAE system of assigning one overall grade per unit to implementing a system where individual submissions receive ratings from 1* to 4*. The unit’s final grade then represented a combination of these individual submissions. This made a massive difference – it wasn’t all or nothing, but, especially where funding was given to one grade and not to the next grade down, this could make a big difference psychologically as well as financially.
Secondly, the addition of impact also made a substantive difference. We were all concerned about it, but actually it became a much more interesting process – much more live to those of us doing it, actually interesting to read, but also much more relevant to people outside the processes. It’s got a lot of political traction because you can tell impact stories in ways that outputs just don’t allow.
REF has always been used to drive policy areas. Open Access would not look like it does in the UK without it being part of REF. Research Councils had policies, but they really didn’t push them – it’s only because REF required OA requirements that it has taken off in the UK.
REF has made institutions take research management and research strategy much more seriously. Unfortunately, institutions have now started to treat it as an end in itself: their research strategy is to do well in the REF or to get a certain grade – and I think that’s really damaging. It should be a means to an end, not the objective.
Pauline Muya: Each subsequent exercise has seen a raft of newer entrants, starting in 1996 with the post-92 universities submitting for the first time. For example, the cathedral universities and the specialist arts colleges are now submitting to the REF. The business and management submission last time had 101 submissions, which wasn’t the case back in 1996.
The post-92s have really established themselves in the process now. I think some of the newer entrants are still struggling, and the balance of panel membership is very often skewed towards the more established institutions. The voice of the small and specialists and newer entrants don’t necessarily get heard when policy decisions are being made.
Institutions manage the RAE/REF process in a much more stringent way than we did in 1996. It never stops – REF preparations just roll over from one to the next, and there are teams in institutions that are there to support it in ways that we didn’t have before.
The whole movement around you don’t enter staff, you make submissions, the decoupling of outputs – that has been quite a big change. Consideration of equality, diversity and inclusion has come in much stronger as well over the last couple of exercises.
Ian Carter: I’d echo Pauline’s observations and would also point to the Stern review as a key development. I think it was a substantive set of changes and it’s taken two exercises to implement the vast bulk of Stern’s recommendations.
Rob Johnson: One question many people are asking, given it’s already been delayed once, is will there be a REF 2029?
Ian Carter: Yes, the current Labour government has made positive noises about needing to have something of this sort. If you don’t have an exercise, how are you going to distribute £2 billion worth of funding per annum? Not having an exercise could be worse in terms of mechanisms that get used. Be careful what you wish for. In policy terms, I can’t see it not happening.
Pauline Muya: I think so too.
Rob Johnson: We are in a fairly unique and challenging environment in UK HE – financially, with drops in international students and huge pressure on budgets. What implications does that have for managing the REF process?
Pauline Muya: Many institutions brought in short-term contract staff in the run-up to REF 2021 to provide extra support. Many of those staff would have left in 2021, and with current financial constraints and REF being so far off, there wouldn’t have been the pressure or priority to replace those staff quickly.
Now that institutions would normally be thinking about making strategic investments in REF support, impact managers, PCE managers, the resources aren’t there. It comes as a double problem – institutions haven’t got the resource to do the work and planning, particularly for something like impact that takes a long time to gather evidence. But they also haven’t got the corporate memory of what we did last time, how it works. Everyone comes in trying to reinvent the wheel with very limited resources.
Yet there isn’t a vice chancellor in the land that doesn’t want to improve their Times Higher league table ranking. That pressure’s there, but without the underpinning resource to deliver what the Vice Chancellors stand up and say that they want.
REF managers are typically around Grade 8 – relatively junior staff propelled into this cauldron of pressure where the vice chancellor wants and expects. That’s a really pressured role for relatively junior colleagues, and if they haven’t got a director or senior support with experience of REF, that can be quite daunting.
Ian Carter: There’s been a transition from having no staff in 1996 – there were no dedicated RAE staff then – to having fixed-term contracts between 2014 and 2021, to now having more permanent teams. In 2012, I was arguing for a permanent team to look after institutional research performance monitoring, not just periodic assessment, because that should be of interest all the time, not just at a REF cycle.
Senior professional service leadership in this space is really important because you (hopefully) understand what will have a material effect. This is only 2025 – there’s 3½ years before the submission goes in. Impact can take a long time to take place and document, but it’s an ongoing process. Sometimes we expect to do too much, too soon. Don’t panic – there’s time to do stuff. You don’t need to have it all in place right away.
Rob Johnson: We’re picking up frustration in the sector about uncertainty, particularly around the people, culture and environment (PCE) element. What are you observing about the current mood?
Pauline Muya: PCE is the bit where institutions are anxious. The pilot institutions are at the sharp end of dealing with submissions at the moment. I haven’t heard anything that gives a really clear direction of travel following the pilot.
My anxiety, echoed by many in the sector, is that we’re now five years into the assessment cycle and we don’t yet know what 25% of the submissions is going to look like. People want to be able to collect data, think about narratives, where to put their effort with limited resources available.
Those that weren’t involved in REF 2014 are probably anxious about the impact and engagement statement because they don’t know what one looks like. It feels like things are happening quite slowly. The main panel chairs have been announced, sub-panel chairs were announced recently, but the sub-panels won’t be in place till the summer. There are still fairly big questions around portability of outputs, staffing left to resolve – people don’t like uncertainty.
Ian Carter: I’m going to take a slightly different tack. The sector always wants to know what the next exercise is going to look like before the previous one was finished, but that’s an impossibility because you want to learn from the previous exercise. My feeling is that we’re no later than previous exercises – in fact, I think the rules have been coming out earlier each time.
The gradual publication of modules sounds good because you start to firm up on areas, but it leaves the harder questions and the interfaces between policy areas unknown. As a systems engineer, I’d say that all the interesting stuff happens at the interfaces, not in the individual black boxes. Until you’ve got the whole lot, you can’t really think holistically.
There’s a tendency of people thinking they want to have absolutely everything really clear, but I’ve always liked to have a framework rather than all the details. If you’ve got a framework, you can work flexibly, whereas if everything is completely nailed down, you’ve got no flexibility.
Rob Johnson: In terms of guidance released so far – initial decisions, Open Access, volume measures, parts of the Code of Practice – has anything surprised you?
Ian Carter: Nothing has surprised me enormously. I’m pleased that the funding bodies haven’t been pushed off course with regard to PCE and the balance between trying to address behaviours and culture in the sector, which is a root issue behind research integrity as well as the working lives of researchers. I hope they continue with this direction of travel; we shouldn’t complain that the PCE pilot is taking time. We want them to do it right rather than rush it.
Pauline Muya: I was slightly surprised that PCE is 25% because that felt like quite a big jump from 15% for environment last time up to 25%. 20% felt about right to me for something that was being ramped up and changed. I know there’s been pushback around the sector about outputs being diminished when they’re the bedrock of what it’s all about.
The big announcement about green Open Access being eligible – well, it always was, so why the big announcement? And there have been things that flip-flopped a bit – talk of the wider pool of staff that could have their contributions recognised, then that came back as “no, we’re not having teaching-only staff outputs and you can’t have PhD students’ theses.” Things like that haven’t been fully nailed down.
Rob Johnson: What should institutions be looking out for over the next few weeks and months?
Pauline Muya: The panels being in place, then the impact and engagement module is due, and the contribution to knowledge module.
Ian Carter: I think impact is really important but probably not going to change massively. The PCE pilot outcomes will be really important, and the contribution to knowledge because that will have to answer the portability question, which is not purely a technical one – it’s a policy and political question for funding bodies to deal with.
In Part Two, we’ll explore practical advice for institutions, strategy considerations, and keys to success in preparing for REF 2029.
To find out more about how we can support your preparations for REF, visit our dedicated site https://www.ref-ready.com/ or contact us at enquiries@research-consulting.com
