Piloting a Monitoring and Evaluation Approach for UKRI’s Open Access Policy

Research Consulting Limited and Sesame Open Science were commissioned by UK Research and Innovation (UKRI) to develop baseline values for monitoring and evaluating the effectiveness of UKRI’s Open Access policy. The primary goal was to assess the feasibility of using open data sources to answer monitoring and evaluation questions related to open access publishing, building on the approach we co-created in 2023 (see: Developing a monitoring and evaluation framework for UKRI’s Open Access policy). 

The analysis focused on peer-reviewed journal articles published between 2012 and 2022, examining levels of open access, collaboration, citations, usage and altmetrics. The team created a dataset by merging information from multiple open data sources, including Gateway to Research, Crossref, OpenAlex and Unpaywall. This dataset enabled the calculation of indicators such as counts and percentages to reflect the state of open access publishing from the perspectives of both UKRI-funded and UK-affiliated authors. 

Our use of COKI-provided infrastructure showcases that operationalising monitoring and evaluation efforts, including a high extent of automation, is, indeed, possible. At the same time, questions about the financial sustainability of open digital infrastructures need to be considered as UKRI moves forward and reflects on whether monitoring and evaluation efforts should be undertaken in-house or through supporting existing infrastructures.