Innovation in evaluation: CYCJ’s learning journey

CYCJ is currently undergoing a six month evaluation to assess its impact on youth justice in Scotland. In the first of her blog series, evaluator Catherine-Rose Stocks-Rankin talks us through the thinking behind this innovative approach, what’s/who’s involved – and why it’s already making a difference.

Asking the tough question: What difference do we make?

CYCJ has taken an innovative approach to understanding its impact. They hired an embedded evaluator – me! – to spend six months working with the team to make sure the gritty and challenging parts of working in the youth justice sector are as much a part of the evaluation as the stories of success. CYCJ champions learning. It is insistent about the changes that Scotland must make to improve life chances for children in Scotland. And in this evaluation, CYCJ wants to ‘walk the talk’ to make sure that it puts learning and development at the core of its own practice.

We began the evaluation in June and I’ll be working part-time with CYCJ till the end of the year. I sit in the office, attend meetings, celebrate birthdays and share in day-to-day office conversations – sleepless nights because the kids are up, older parents who need care, holidays, new projects, the excitement about going back to Uni. These are just some of the things we’ve shared as colleagues.

But of course, I’m also ‘the evaluator’. While the team is out and about supporting people in front-line practice, or helping to implement the Youth Justice Strategy or creating space for young people to come together – my work is focused inward, on the organisation and its impact to date. The work I do is to help the team grapple with their own practice, to see where they make a difference and where their work can be strengthened and amplified.

In many ways, I’m doing the work that CYCJ does for the sector. I’m supporting them and I’m challenging them. I’m here as a resource on evaluation and knowledge mobilisation. But I’m also a reminder that things can always be improved. Where there is evidence to validate their work, I’ll offer it. And where there are insights that can help them learn, I’ll make sure those messages are practical and clear.

Generating evidence with learning and action in mind: Why trust has to come first

We know that evidence is more impactful when users trust it. And we know that building a relationship with the users of evidence is the best way to build trust.

The CYCJ team – and the Scottish Government as their primary funder – need to trust that I have worked hard to understand them. They need to trust that the findings I’ve produced are robust and credible – that I haven’t shied away from difficult conversations or skimmed over the more challenging stories. Trust is crucial. Trust is what makes it possible for us to learn and change. Without trust, we can all find a reason to reject the messages we’re given and do the same old thing again and again.

How do you build trust in evaluation?

Trust is not something we ever really ‘achieve’. It’s never done. It’s a process and it takes work every day to build and maintain. Even though this is a short evaluation project (six months isn’t long to understand an organisation like CYCJ!), trust was on the forefront of my mind from the first day and it will be to the last.

Here’s what I do to build trust:

1. Lead with values
2. Use a double diamond to speed learning
3. Show my work
4. Have fun
5. Be honest and say what I don’t know

Values: When I began the evaluation, I had conversations with team members as well as the Scottish Government’s Youth Justice team. I shared my values for this project – which are: collaboration, transparency, flexibility, appreciative and ethical.

Double diamond: I use the UK Design Council’s Double Diamond as framework for project management. I find that it helps me know when I’m in an expansive learning space (doing interviews or focus groups) and when I need to take that learning and turn it into formal product outputs.

Show my work: I’m doing a bit of that right now in this blog! Here’s what I share with the CYCJ team, the Scottish Government and the project’s Advisory Group: the project plan, the methodology, the meeting minutes from the project advisory group, initial reflections, interim findings – even some of my personal reflective practice.

Have fun: CYCJ staff are tremendously hardworking, but they also like to have a laugh. You may know this already, but the team is very generous. I turned 40 this summer and they helped me celebrate. That’s not something you see all that often in evaluation dynamics.

Be honest: I am not an expert in youth work or the justice system. I’m an expert in evaluation and knowledge mobilisation. I’ve had to ask to some pretty basic questions about how the system works. And I’ve had to do a lot of listening and learning. As I should. Being humble is the best way to do research. I embrace it!

What will this project achieve?

This evaluation is producing four core products:

1. An evidence bank about CYCJ’s role, its reach, and impact
2. A six-month learning process which will inspire and challenge the team
3. A report which documents CYCJ’s impact
4. An event for the youth justice sector to celebrate where positive change has occurred, and challenge ourselves to push further

The evaluator’s toolbox: A note about methods for those who like to get to the detail of the thing

This evaluation uses:

• A theory of change approach to show contribution – not attribution
• Appreciative inquiry to prioritise learning
• Participatory methods to ensure robust and ethical consent from CYCJ team
• Embedded role for the evaluator to build trust with CYCJ and its stakeholders

I use a theory of change approach to evaluation called contribution analysis. Theories of change are useful because theyshow the pathway to impact. In Scotland, we’ve really embraced a culture of contribution over attribution in social services. This shift is part of the larger culture of outcomes-focused practice. There are lots of resources on contribution analysis. I’ve even produced a few of them myself. Have a look at this literature review for a start!

I’ve improved the theory of change approach over the years by making it appreciative – which means I put learning and deliberation center stage. Appreciative inquiry is a powerful way to ensure that our minds – and our hearts – are open to change and development. Appreciative inquiry allows for camaraderie and even fun (!) in the research process. Having a bit of fun where we can, laughing when it’s possible, is fundamental to making sure our brains are open to learning. I enjoy having a laugh of course. But my end game is to make sure that evidence and the learning really sticks.

This is a participatory project – which means I am explicit about when and how I collect evidence. I use workshops with the team and collect evidence through post-it notes and discussion to make sure everything is out in the open. I am clear about when I’m taking research notes and ask permission. I use my own experience of being embedded in the team to inform my own weekly reflective practice. This keeps me ethical and honest – and it helps me learn.

I’m embedded in CYCJ. Being an insider and outsider offers a special vantage point on the organisation. I’m close enough to the team so that I can develop a nuanced understanding of their work, but enough of an outsider to ensure that I act robustly and ethically as a researcher. I use the KOLB learning cycle to track my reflections each week. It helps me keep on track of my own learning as a member of the team – and keep me straight ethically so that I honour my commitment to be both inside and outside the team.

A final thought

Too many evaluation reports sit in a desk drawer, un-read. Or worse still – they never even get downloaded. That email just sits in an inbox and one day gets deleted. I can’t imagine a worse way to use the precious reserves of a third sector organisation like CYCJ, the tax-payer’s money much less my own time. We’re doing something different at CYCJ. Keep up date on our progress by joining the CYCJ mailing list or following them on Twitter. You can also follow me to see my own reflections on the process! And look out for our event in 2020 where we’ll launch the report from this work.

About our blogger

Catherine-Rose Stocks-Rankin is a researcher, knowledge mobiliser and evaluator. She has expertise in both the theory and practice of knowledge translation and evidence into action. She describes her methods as ‘creating a bridge between policy, practice and community with a focus on generating evidence that leads to meaningful action and social change.’ Read more.