Developing a monitoring and evaluation framework for UKRI’s Open Access policy

Following the publication by UK Research and Innovation (UKRI) of its Open Access policy in August 2021, Research Consulting was commissioned to support the development of a monitoring and evaluation framework. The intention of the M&E framework is to assist in assessing progress towards Open Access, levels of compliance and the effectiveness of the UKRI policy.

In addition to in-depth desk research exploring existing monitoring and evaluation approaches, we engaged with 76 stakeholders, through interviews, focus groups and a workshop. Contributors included experts from both the research and publishing communities, shaped by UKRI’s commitment to the principle of co-design in developing the framework. The findings from the stakeholder engagement were used to outline five principles for monitoring and evaluation in this area:

• Focus on policy improvement, not just compliance
• Build a balanced and inclusive overview of the publishing landscape
• Craft clear and unbiased questions
• Operate transparently
• Seek efficiencies and minimise burdens

Working with associates Bianca Kramer and Cameron Neylon, we developed a longlist of over a hundred potential M&E questions. This was refined to a shortlist of 20 questions, including both quantitative and qualitative elements. A detailed data specification was created for each M&E question, suggesting recommended approaches to answering them, potential data sources as well as aggregation and analysis methodologies.
The recommendations from this project will be considered by UKRI as it develops its final monitoring and evaluation framework for its Open Access policy.