- Quench Arts
Measuring Impact
As we pull together our final report for our organisational development project we thought it’d be timely to write a blog about why we at Quench Arts place such a huge focus on measuring the impact of our work.
If you’ve ever worked for Quench, then you’ll know that we have high expectations in regards to levels of feedback and communication from our music leaders and partners – this is built into all of our contracts and our induction and training for staff and we feel that it is a crucial element of the role. On our wellbeing programmes we collect the following information:
Sessional reports for all individual and group sessions. These include feedback from participants with a jointly agreed focus for the next session; a summary of the session content; any relevant music leader or staff observations against the project outcomes; any concerns or actions needed. At the same time as sending in these reports, we receive any audio recordings taken of the musical outcomes plus audio from group reflection discussions, photographs of lyrics and any graffiti wall or ‘post it’ comments from participants.
Additional sessional monitoring and reflection feedback as devised with the project participants and team. Some examples include photos and quotes from our ‘metaphor reflection box’ exercise, where participants chose an object from a box and explain why they’ve chosen this to explain how they feel at the time; numbers from our ‘confidence counters’ exercise, comparing how the group collectively feel at the beginning and end of the session (by posting counters in boxes); photos of our ‘dart board’ where participants place a velcro dart on the board reflecting how well we are achieving the statement or aim written alongside; or written feedback from our comments post box/starter questions, etc.
Participant baselines. These are a series of questions and exercises near the beginning of the project whereby participants self-reflect where they are on a scale against a series of set statements. Their answers are then revisited at various points in the project and at the year-end to measure progress. This data is analysed on a cohort basis to give statistical information (e.g., 70% of participants felt that they have improved…., etc) but also used to inform individual participant case studies, giving more of a personal picture of progression. Our baselines normally have 20 statements but these do vary slightly with the aims of the project and the particular outcomes that we are hoping to achieve. The statements cover areas such as what participants feel about their: musical abilities, skills and opportunities; self-confidence and self-esteem; health and wellbeing; and their social engagement. For each section we tend to utilise elements of established validated scales as most relevant to the particular project (for example, Youth Music’s Musical Development Scale; The Warwick–Edinburgh Mental Well-being Scale; The Outcome/Recovery Star; The Rosenberg Self-Esteem Scale, etc). The final section of the baseline is where participants set personal aims for their project engagement alongside each of the ‘Five Ways To Wellbeing’. Depending on the participants’ wishes and needs, this information can be collected creatively through practical tasks led by our musicians (moving a mixing desk fader, or example), documented through a conversation or completed as a paperwork exercise. As well as recording the data of where the participant is on the baseline scale for each statement, we place a real focus on finding out the reasons for their choice – often the conversation around any movement is more important than the movement itself. With mental health, a participant might be hugely affected day-to-day by events that have recently happened, medication, etc, and so gaining the context is incredibly important. For example, a lower score regarding participants’ self-assessed musical ability at the end of the project (compared to the beginning) might not always be a negative thing as they could have potentially been motivated and inspired by other more experienced musicians and now realise how much more they could progress.
Monitoring/progression data from any partners/professionals involved with our participants, particularly when we are working with inpatients in secure or acute settings. As well as detailed risk assessment information at the start of the project, information such as MOHOST data from Occupational Therapists (Model of Human Occupation Screening Tool) and Pupil Progress Profiles from Hospital Schools can help to back up progress and observations seen in project sessions and also give a wider context to our work.
Participant, music leader and partner end of year evaluations (minutes from meetings as well as individual questionnaire feedback). This information not only helps us to further assess the impact of the project but also to consider and reflect on any suggestions for improvements in project delivery for the future.
Participant case studies. These are normally written by our project music leaders with additional input from Quench Arts Directors and we always try to include comments and feedback from parents/family and professionals involved with the participant to get a wider context. These case studies aim to give a clear picture of what the participant’s life/situation was like when joining the project and how it has changed at the end of the project year, pointing to evidence of this change taken from all of the monitoring and evaluation materials collected above. They summarise and give context to all the other data and feedback collected.
Music Leader practitioner baselines. Similar to participant baselines, these are used particularly for our emerging and developing music leaders to assess their practice at the beginning of the project and revisit this at the end to identify progress. With these baselines we also ask music leaders to set their own aims for their practice development for the duration of the project year, using Youth Music’s ‘Do, Review, Improve’ quality framework.
Music Leader Peer Observation reports. At Quench Arts we have established a peer review process, where staff are paired with a different music leader for each project year and each visit each other and observe their practice on a self-identified specific area or criteria, using the quality framework detailed above. Each pair learn from observing and discussing their practice and strengths/weaknesses. Staff have found this really helpful as they may often work in isolation and it also helps us address any collective professional development/training needs.
So, why do we collect this information? The most common answer to this question when we ask it is ‘to prove the value of the project to funders’. Naturally, that is definitely an important reason, as organisations supporting the work always like to know that their money is being spent appropriately, is valued and is helping to support positive outcomes. However, below are some other key reasons:
To help build a positive relationship between participants and their music leaders. The more our staff know about each participant, their aims for taking part and their particular musical interests, the better they can plan and tailor activities to their needs.
To make sure that the project is running smoothly and to address any issues as they arise. This is especially important when working in the mental health sector, where it might be inappropriate for the project manager to attend project sessions early on in the project when an extra, unknown adult in the room for, say, a one-to-one session, might affect engagement. No-one can solve a problem if they don’t know about it! We always encourage our music leaders, partners and participants to be completely honest in their feedback so that we can address any problems and support however needed. Sometimes this might mean discussing solutions as a team and making informed changes to the programme delivery or content. If a participant is making less progress than expected or not engaging well, it might mean seeing if they respond better to an individual music leader in their individual sessions, or moving to pairs work or a different musical focus.
To help us improve as an organisation and make sure that we continue to develop our practice moving forwards. We understand that projects might not always end up running quite as originally planned for various reasons. We always listen to feedback from our music leaders, participants and partners to see what isn’t working as well as what is, to inform future delivery.
Because finding out about the positive impact of the work and sharing this is incredibly important motivation for the whole team. Work in this area can be challenging at times and most people in this line of work tend to be sensitive souls who always aspire to do better, often being quite self-critical in their reflections. Hearing about how a project might have changed someone’s life for the better, helped them make friends/go to college/get out of bed or leave their room for the first time can be a real boost and inspiration to keep at it. This can especially be the case for project managers and administrators who often mainly hear about problems rather than successes, don’t always get to see first-hand any changes and might not always have built individual relationships with participants - reading how the project has made such an impact for participants can make the hours of funding applications and budget management seem even more worthwhile.
To prove to partners, statutory and health organisations/managers the value of the project, both at a statistical cohort level and on an individual level. This can help with strategic profile and assist in building a sustainability strategy for the work, potentially also encouraging partners to consider embedding similar activities in their own offer in future.
Finally, and for us this is the most important reason, we think that collecting all the information above is really important in giving participants ownership over their participation. They can set their own aims and targets for the project and can also reflect for themselves where they are at the moment and how they want things to improve. This can be really empowering for participants. Often people find it hard to see how they are improving on a day-to-day basis, so having evidence and information back from the start of the project can help participants see the journey that they have made. This can really highlight their progress and success, building self-belief and pride and increasing motivation to further engage and progress!
