In this blog, ESS Board Member Katherine – Head of Research & Evaluation at Scottish Book Trust – shares how evaluation is approached at SBT.
At Scottish Book Trust we run a range of programmes to help people access the life-changing benefits of reading for pleasure and express themselves through creative writing and personal story. From our Bookbug programme for Early Years to our Reading is Caring programme for those living with dementia, via Book Week Scotland and the New Writers Awards, we really do offer something for everyone in Scotland!
A lot of our work is delivered at arms’ length in partnership with schools, libraries, health professionals, early years settings and many more organisations, meaning we are rarely ‘in the room’ to collect data from our ultimate beneficiaries. For example, we provide training and resources to learning professionals to support them to embed a reading culture through our Reading Schools programme, but we aren’t able to talk directly to pupils in all 495 (and counting!) Reading Schools about how taking part has affected their attitudes to reading.
Our approach to handling this complexity is to build theories of change for each of our programmes.
These help us to map out our programmes in detail, articulating how exactly we need to design and deliver that training and those resources in order to make it most likely that pupils will go on to develop the love of reading that research shows supports attainment, wellbeing, critical thinking, creativity, empathy and resilience.
From our theory of change we then create an evaluation plan, pinpointing the key short–medium term outcomes and identifying indicators and methods to collect evidence around these. This allows us to measure the things that matter (as well as, in a practical sense, the things that we can) within a strong framework that demonstrates how these more immediate outcomes contribute to that longer-term impact.
As well as publishing evaluation reports, we also hold internal reflection sessions, helping us to ensure our evaluations are both useful and used. In these the team get together to discuss key learnings and any changes needed to the programme to increase impact; we also evaluate the evaluation (!) by reflecting on how well our plan worked and what changes may increase engagement and rigour next time.
One area for development for us at the moment is working on including pupils’ voices more strongly and more meaningfully in our evaluation.
For example, in 2021 we held comic workshops with Paul Bristow, one of the fab authors we work with regularly, to gather pupils’ views about our book gifting programmes. This approach meant that pupils got to work with a brilliant Scottish author and take part in a fun activity, schools got a fully-funded author workshop, and we got some detailed and fascinating feedback about the impact of our programmes from the people we most wanted to hear from – a win-win-win!
Overall we’ve found that using a theory of change methodology has really helped us to focus on the long-term impacts we aim to support and think about exactly how our work needs to be designed and delivered in order to achieve these. This gives us a strong framework for evaluating our programmes and improving them year-on-year, but also allows for flexibility across our wide-ranging programmes and audiences.
Our principles for good evaluation can help you to judge if you are happy with your evaluation. They are used within this blog to highlight the principles that are being demonstrated:
– Good evaluation should be about what matters
– Good evaluation should be used by you
– Good evaluation should involve the people you support
See below for some useful resources related to this blog:
Support Guide 1a – Setting outcomes
This guide helps organisations work out what their outcomes are and how to write them.
Support Guide 1b – Working out what to measure (Setting indicators)
This guide explains ways to develop and use indicators to help measure outcomes.
Support Guide 1c – Developing a Logic Model
This is a practical guide that focuses on how you put together a basic logic model to help you think about your aims, outcomes and activities.
ESS Evaluation Tool: Interactive Logic Model Template
This interactive logic model has been designed as an editable electronic document which you can save, edit, share with your colleagues and print.
Introduction to logic modelling online learning
The video will help you build your own basic logic model, use a logic model to link your work to higher level outcomes, and use a logic model for evaluation.
Theory of change in ten steps
NPC’s guide to developing a theory of change for charities and funders.
Creating an evaluation plan
ESS Evaluation Support Manager Nicola Swan talks about why having an evaluation plan is useful and how to make one.
