Between August and November 2022, almost 80 individuals from across the research and publishing landscape contributed to a study we delivered on behalf of UK Research and Innovation (UKRI), to support the development of a monitoring and evaluation (M&E) framework for their Open Access (OA) policy.
The framework will help UKRI and the sector assess open access progress, levels of compliance with the policy and its effectiveness. It will also seek to establish insights into open access publication trends across the UK and, where possible, their impact on academic practices and society.
We are in the process of finalising project outputs for public dissemination alongside our associates Bianca Kramer and Cameron Neylon, but we are now in a position to share some high-level findings and next steps. This blog covers five key principles we identified from our discussions with the research and publishing communities, as well as considering the implications for UKRI’s future M&E efforts.
Focus on policy improvement, not compliance
Contributors strongly recommended that monitoring and evaluation efforts focus as far as possible on identifying positive lessons and areas for improvement, for example around policymaking, policy communication, policy implementation and awareness-raising. This is in contrast to using M&E results solely for activities such as identifying non-compliant researchers or institutions.
We also discussed the idea that constructive and positive discussions about policy compliance, alongside recognition of successes in broader OA activities, could encourage and assist the spread of best practices. For example, institutions are keen to understand how others have approached challenging areas and to learn how these solutions may be adapted to their local context.
The above suggests that UKRI’s M&E efforts could helpfully stimulate discussion around OA and the sharing of good practice in the UK and internationally. We note that this is not only from an institutional perspective, as the insights emerging from M&E results may well support improved policymaking and planning across other stakeholder groups such as research funders and publishers.
“I think anything that is monitored, ought to perhaps be done with a view of communicating back on the positive side of this, rather than being seen as an exercise of compliance.” – University Library
Build a balanced and inclusive overview of the publishing landscape
There was recognition that M&E questions about articles are easier to address compared to long-form outputs or broader trends in the OA landscape, mainly owing to the maturity of data sources. Contributors supported UKRI’s plan to consider these more complex areas in their M&E efforts, even if results may be limited and should be caveated appropriately.
Considerations around diversity extend beyond research outputs, however, and several contributors commented on the importance of considering organisations of different scales – for example, large research-intensive universities and small and specialist ones, or large international publishers and small national, specialist or institutional presses.
This distinction is important because larger and more structured organisations are likely to have more comprehensive and established OA-related workflows, while smaller ones may rely on manual/ad-hoc processes and analysis to a larger extent. As a result, one should remain aware that data on (and from) different types of organisations may introduce bias arising from their circumstances rather than from UKRI’s OA policy itself.
To address these challenges, UKRI will need to consider data sources that are as comprehensive as possible, as well as to ensure that the narrative accompanying M&E results appropriately acknowledges the limitations and inherent bias of data.
Craft clear and unbiased questions
Contributors emphasised the importance of designing unbiased M&E questions, using clear and unambiguous language, and in a number of cases offered to sense-check or test definitions. This provides a route for continued stakeholder engagement and co-design of the M&E framework, which UKRI are keen to deliver and implement as the continuation of a highly collaborative process.
I think what is very important is clear terminology and clear, simple questions… because there seem to be a lot of instances where people think they’re talking about the same thing, but actually have a slightly different understanding of what [question] they’re answering. – Publisher
As part of this work, we have been building on consultation findings and extensive desk research to develop a series of recommended questions to assess the effectiveness of UKRI’s OA policy. The final outputs will include these questions as well as a recommended approach to answering them by (mainly) using open data and open infrastructures.
UKRI anticipate that the design and operationalisation of the M&E approach will be an iterative process, including further refinement of questions where needed, to ensure they are clear and unbiased.
Operate transparently
Contributors broadly agreed that, when sharing M&E results, UKRI should be open and transparent about the methodology followed and any known limitations of the approach. Open sharing of data gathered in the M&E process was widely supported and felt to be consistent with both the spirit of open research and the need for transparency and accountability regarding use of public funding, in line with UKRI’s original intentions.
Contributors mentioned numerous ways in which openly available M&E results could be used, such as raising awareness of OA, informing negotiations between publishers and research organisations or assessing any unintended consequences of the approach to OA funding.
In the coming months, UKRI will be considering any limitations in terms of data sharing, particularly in cases where M&E questions can only be addressed by the means of organisational or licensed data. We do not expect this to affect the M&E approach significantly, however, and acknowledge that a large number of the M&E question areas can be addressed via open data.
Seek efficiencies and minimise burdens
We note a clear desire to ensure that UKRI’s M&E approach builds on existing efforts and initiatives, rather than duplicating efforts. There is an opportunity to liaise with other funders as well as relevant international projects to identify efficiencies and areas for collaboration, but we also note that UKRI’s efforts will be specific to their own policy and will therefore need elements of ad-hoc setup and analysis.
In terms of sharing between [organisation] and UKRI, I would like to see as much cross-working and sharing as possible, not only on specific data but especially on methods to ensure as little duplication of effort as possible. – Research funder
Contributors from research-performing organisations noted that reducing reporting burdens is key. In this regard, we note the following:
- In terms of monitoring the compliance of research-performing organisations with UKRI’s OA policy, findings to date suggest that this can be addressed using existing data sources, with no need for new reporting requirements as a condition of funding.
- In terms of broader M&E questions around the effectiveness of UKRI’s OA policy, it is important to consider that these will necessarily have a qualitative element, to allow for nuance and discussion. As a result, UKRI is likely to require some input and engagement from a sample of research-performing organisations. In line with UKRI’s commitment to reducing unnecessary bureaucracy, there will be a focus on streamlining any additional requests for input and, particularly, on identifying opportunities to integrate these with existing activities.
The above will be carefully considered in the recommended methodology to answer M&E questions, which will be made available as part of the final project outputs.
What happens next?
We would like to thank all the individuals and organisations who have contributed to this work so far. The breadth and depth of views provided were extremely useful and are a testament to the community’s interest in and commitment to open research practices.
As noted above, we are now in the process of finalising project outputs. These will comprise a final report with recommended M&E questions, an executive summary and a data specification outlining a recommended approach to answer the proposed M&E questions. UKRI expects to share these documents in the summer of 2023 via the Zenodo repository. Further communications will take place via the UKRI website as well as on Research Consulting’s and UKRI’s social media channels.
If you have any questions about this work, please contact openresearch@ukri.org.
