Here at ESS we’ve been doing our best to support third sector organisations who find themselves facing new evaluation challenges in light of the Coronavirus lockdown. For many, the way they deliver their services (and even who is in need of their services) has changed – meaning that the way they evaluate has likely needed to change too. In this blog, Training Officer Shona Wells explains how ESS has been adapting its own services (and, therefore, evaluation processes) in order to respond to this sudden new normal.
The first thing we were keen to do when lockdown began was to make sure the sector knew that ESS was (and is!) still here. We set up our COVID-19 webpage and are regularly updating it with advice, resources and online events, so that we can keep supporting organisations to self-evaluate. It might seem daunting, but even in these challenging times organisations can (and should) evidence the difference they’re making and capture what could be some very valuable learning. They may even find that some of the changes work well and they’ll want to keep them after lockdown ends!
In order to keep helping organisations with self-evaluation, we have created a virtual version of our Let’s Evaluate! training programme, which ESS Training Officers are delivering now. We always try to make our training fun and interactive and, as we hadn’t delivered the programme online before, we had to think hard about how to do so without losing these important elements. So, with lots of research and practise with ESS colleagues, we devised our current Let’s Evaluate! online programme. Here’s what we’ve learned so far:
- Online training should be delivered in shorter periods
Usually the programme takes the form of three full-day workshops, spaced a fortnight apart; the new content is delivered via Zoom video conferencing, with each workshop split into two shorter sessions that are delivered weekly. This felt like an important adaptation to make as, after just a few weeks into lockdown, most of us started to realise that Zoom-fatigue (or being ‘Zoomed out’, as we call it) is a real thing!
- Take time to consider accessibility
We had to consider the accessibility of our new workshop format. Participants are emailed all handouts in advance, which are numbered in the order that they’ll be opened during the workshop. We’ve converted the handouts to an editable PDF format so that participants can type straight on to them and save their work. We also discussed the new workshop format with a deaf participant to make sure it was going to work for him and his two British Sign Language interpreters – which, so far, it has, as he has been able to ‘pin’ the interpreters’ videos on Zoom so that they stay on his screen throughout.
(You can see the interpreter in the second row, rightmost thumbnail of this Zoom meeting screen grab)
- Make slides visually (more) appealing
We’ve tried to make our PowerPoints simple yet visually appealing, given that the screen-share of the slideshow is the main thing that participants can see (as opposed to the real human interaction they usually get in the workshops).
- It’s important to work out the mechanics of managing a group
Given that there is more scope for participants accidentally talking over each other in an online workshop, we have also been asking them to keep their microphones muted and ‘raise their hand’ (virtually!) when they have a question or something to contribute to the discussion. This means that it’s easier to hear the facilitator during theoretical explanations, and other participants during groupwork, feedback or questions.
(We have been using Zoom’s ‘Polls’ function for workshop activities)
(We have also been using Google’s Jamboard as virtual post-its)
- Different delivery formats require different ways of evaluating
Usually we would evaluate the workshops by asking participants to complete and hand in an ‘after’ questionnaire, so that we can compare with their ‘before’ responses their understanding of the learning outcomes, and so that they can tell us about anything else that did or didn’t work for them. We have adapted our questionnaires to an online format using Google Forms, and we ask that participants complete these before leaving the Zoom workshop (rather than sending them the link and asking them to do it after – because we know that we might not get everyone’s responses that way).
- Permission to collect photographic evidence
We would usually ask for people’s consent to take photographs during events, as another way of evidencing our impact. In the new virtual format, we do this by asking participants to tell us in the Zoom chat function whether or not they are happy for us to take screen-shots.
- Use feedback to learn, improve your services, and celebrate
Participants were also invited to use the chat function to give us any additional feedback after our first Zoom training event (part 1 of Getting Started: Outcomes and Indicators), which gave us some useful tips that we’ll use to make our workshops even better next time. No-one gets things right all of the time, especially when switching from face-to-face to online delivery as many of us are doing now; our advice is to use the feedback you are given to learn, to improve your services, and to celebrate success!
“Thank you for today, I didn’t know how I felt about learning online but it worked really well” (Fiona Taylor, SPFL Trust)
“I was very pleasantly surprised at how well the training worked online – a really informative session. Great job, well done!” Kirsteen Powell, Carers of East Lothian
If, like us, you find yourself having to adapt your services and find new or different ways to evaluate, you may find it helpful to check out our COVID-19 webpage for tips and resources. You can also join our mailing list to be kept up-to-date with any new advice we have to share or events we’re facilitating.
