As CYCJ’s evaluation exercise approaches its conclusion, evaluator Catherine-Rose Stocks-Rankin shares her thoughts on what she’s learnt about CYCJ, the youth justice sector and the process of evaluation itself.
Good evaluation requires humility
For me, evaluation is first and foremost about learning. I have a tested set of methods that I use, but every time I do an evaluation – I have to start at the beginning and do the hard work of admitting that I know almost nothing. And that’s the way it should be.
When I began this work, I knew almost nothing about youth justice sector. I told the CYCJ team that I was an expert in organisations like theirs – boundary spanners that cross many worlds of academic, policy, practice and lived experience. But I’m no expert in youth justice and I had to say so.
Evaluation, done well, is as much about exploration and deepening understanding as it is about developing robust evidence. Evaluation needs an openness and it needs a vulnerability. Otherwise there’s no space for learning.
What’s been wonderful to see is that others have been able to open up to me. CYCJ has been open about its work, and its struggles. The 24 members of the youth justice sector that I interviewed were just as candid about their challenges and where they wished their work could go further.
Good evaluation includes a robust understanding of context
I learned a tremendous amount about the sector. And because of that learning, I am able to be make conclusions about CYCJ’s role, and impact, within it.
Here are some of the rich insights people shared with me about their work:
I learned that the majority of professionals working in the sector feel there has been immense progress – from punitive approaches and outputs-focused practice determined to “reduce offending” towards a more trauma-informed workforce which looks to understand underlying issues and support young people and their communities.
But I also heard professionals within the sector describe their sense of precariousness. The majority of interviewees talked about a sense of feeling marginal. For practitioners, this might be a sense of feeling marginal within a larger Children and Families Team and somehow irrelevant to the Adult Criminal Justice Team. Even those working in Academia talked to me about feeling marginal as practitioners turned academics.
The precariousness was a powerful theme across these interviews. But, as one of the Advisory Group for this project put it, precariousness isn’t all negative. It can bring an “energy” around the work. This seems very true, there is a sense for so many interviewees that working with young people is a “life’s work” – that the change they want to see will take time.
Good evaluation provides evidence of day-to-day work – as well as impact
And most importantly, given the work I was commissioned to do, I learned about CYCJ. I learned about their reach, not just across Scotland, but across the different parts of the youth justice sector, from social work to the police to the Scottish Prison Service. They reach young people with justice experience and the academics and researchers who work with this group to monitor trends and investigate best practice.
I learned about CYCJ’s role in working across the breadth of the sector and within the hierarchies of authority in the youth justice workforce. I learned about where CYCJ’s resources are used in the university classroom or as intel for Ministers. I learned about CYCJ’s work to support practitioners working directly with young people with argumentation to help them make the case for innovative forms of support. And I learned about CYCJ’s work to support service managers with culture change.
These are just a small taste of the learned we’ve done together and I can’t wait to share the rest when we launch the final report in February.
Good evaluation is about learning, and learning requires trust
When I wrote about the innovative approach we were taking with this evaluation, I emphasised the importance of building trust. I think CYCJ and I have come further than they might have expected down that road. I think the humility we’ve shown each other is a large part of that trust.
CYCJ has been willing to learn from me – from my approach to evaluation and my focus on reflection. And I’ve been open with them about what I didn’t know and where there have been gaps in the research. If you really care about trust, then you need to be prepared to hear the challenges when they come.
I’ve tried to myself available to CYCJ – and to the Scottish Government – so that I can hear where the research resonates, and where it needs more depth. And this is what I hope they’ll offer back; a willingness to hear challenge when it comes, and enough trust in me and the process we’ve been through to build on the learning.
However, don’t just take my word for it! I asked the team for their thoughts on this process and they responded with pretty interesting – and surprising – feedback. Read more here.