If you’re running a project or programme, the chances are that some form of evaluation is going to be needed, whether that’s to gather feedback for improvement, meet a funder’s requirement, or support a case for continued funding.
Usually, once a programme is up and running and delivery is progressing, attention finally turns to evaluation. At this point, research managers, evaluation leads and professional services teams start to ask questions like what do we want to know? How will we measure success? What evidence will we need?
In our experience of running dozens of evaluations ourselves, we’re sharing three pieces of advice to help you prepare for evaluation.
1. Prepare early
Ideally, think about evaluation while you’re still designing the programme or intervention. However, we know that’s not always possible as capacity is stretched during design, and evaluation often isn’t prioritised until evidence of impact is needed.
One of the most useful things you can do at an early stage is build a theory of change, helping you map out what your programme or intervention is doing, why, and what you expect to see change as a result. A good theory of change can also be adapted as the specifics of a project become clearer, while still giving you something concrete to test rather than a general question of whether the programme “worked.”
The Magenta Book (the UK Government’s official guidance on evaluation) frames this well: a theory of change is a planning tool that exposes “the assumptions upon which the intervention is based” and supports evaluations that are “proportionate” to the resources available.
Co-creating the evaluation with the people closest to it matters too. We took this approach for Wellcome’s evaluation of a COVID-19 open data-sharing statement, developing the theory of change collaboratively with the funder and an advisory group before the evaluation began.
That gave us a specific set of hypotheses to test, not just a general question of whether the statement “worked”, and let us track, hypothesis by hypothesis, how strong the evidence for each one was. We took the same approach on two evaluations of projects funded through Wellcome’s Institutional Funding for Research Culture, where the theory of change guided the whole approach from the outset.
It is also helpful to set out your baseline early and think about what “now” looks like, so you have something clear to measure change or impact against. That often starts with your own organisation’s records and thinking about what you already hold from before the programme began. But a useful baseline doesn’t need to be a perfect dataset – it’s worth being realistic about what you can actually measure. Do you have the information you need, is it reliable enough to use, and will you still be able to collect it the same way later? It’s also worth asking whether the programme is likely to run for long enough, or be large enough, for meaningful change to actually show up: a KPI might be easy to count but tell you very little about whether the programme made a difference, while an outcome that matters may be unlikely to shift measurably within the evaluation’s timeframe. Working this out early gives you the chance to fix your measures before the programme gets underway, rather than discovering at the end that you have no reliable picture of where you started.
None of this assumes the programme will unfold exactly as designed. Programmes often shift as scope evolves and delivery adapts to circumstances. What you’re setting up early is a reference point to return to as delivery progresses.
2. Gather structured evidence
Once you know what outcomes you’re looking for and understand the data available, it’s worth thinking about who you need to involve in your evaluation, and when. A useful starting point is to think about the “process” and “impact” elements of your evaluation separately — the Magenta Book takes the same approach.
A process evaluation, exploring how a programme or intervention was actually delivered, is often easiest with people who’ve recently been through it, while their experience is still fresh and specific: their experience of a training programme, say, or an application process. That’s often best done early, or during delivery itself. Understanding impact, on the other hand, often requires following a cohort far enough beyond the programme for meaningful effects to emerge, which usually means waiting until later, once things have had time to bed in.
Process and impact evaluation often need different populations, too. Process evaluation is often best explored with people who’ve recently been through it, such as a training programme or an application process, while their experience is still fresh and specific. Impact evaluation, by contrast, usually needs a cohort far enough on from the programme for any real effects to have had time to show up.
3. Be ready to learn what you didn’t expect
An evaluation done well can highlight findings that you anticipated and those that you didn’t. For research managers and funders, that means being open to criticism it might uncover, and seeing it as an opportunity for improvement rather than a verdict.
One evaluation of a programme supporting industry co-funding for doctoral training found the programme was shifting behaviour and strategy inside participating companies, a genuinely unexpected benefit, well beyond what the funding had set out to achieve.
Another, for a funder, surfaced ongoing frustration in how the lead university and its partners were working together. It was a problem entirely out of the funder’s sight and beyond their direct control, but one they needed to know about, so they could see this as an opportunity for improvement in future funding rounds.
Both of these findings came from asking open questions, and being willing to hear an answer you weren’t looking for.
Before you start your next evaluation
Each of these lessons comes back to the same thing: judgement. Applying an evaluation framework is relatively straightforward; the harder part is deciding what you need to know, what evidence will answer those questions, what context you need to understand, and what you will do with the findings once you have them. Those decisions matter most before data collection starts, when you still have choices about the questions you ask, the evidence you collect, and the people you involve. Leave that thinking until later, and the evaluation can end up answering the questions that were easiest to ask rather than the ones that matter most.
The earlier you make those decisions, the more useful, and usable, your evaluation is likely to be.
If you’re considering an evaluation, or want to talk through any of this, do get in touch.
