Using program evaluation to strengthen service delivery: a case example of Open Minds CHIME program
If you are an organisation seeking to demonstrate your impact, enhance service delivery and better position yourself for future investment, do not go past two critical pieces to the puzzle - monitoring and evaluation.
Beacon Strategies case example
CHIME: evaluation in action
A few years back, we were engaged by our partners at Open Minds to undertake an evaluation of their CHIME program.
Open Minds is a leading not-for-profit provider of mental health, disability, residential and social support services. Established in 1912, Open Minds has over 400 staff working across 23 locations in Queensland and Northern New South Wales.
CHIME is a program designed by Open Minds that supports people experiencing severe mental illness with complex needs to live in their community. The program aims to promote recovery and independence for clients through the principles of connectedness, hope, identity, meaning and empowerment (CHIME).
Skilled and experienced staff partner with local hospital services to provide integrated clinical and non-clinical care for people being discharged from mental health inpatient facilities. Eligible clients are offered case management and care coordination to help support their mental health recovery needs within a person-centred model.
CHIME has been delivered since 2014 in the Sunshine Coast, Gympie and Mt Isa and is currently funded by Queensland Health until mid-2019.
What did we do?
We undertook our evaluation of the CHIME program using a mixed-methods approach, consisting of:
quantitative client data analysis of psychometric measures over a 12-month program sample
qualitative data analysis and theming through semi-structured interviews and focus groups with past clients, program staff and stakeholders from the local hospital service
triangulation to position findings within the broader context of evidence-based and emerging models of practice and relevant policies and frameworks
developing a comprehensive evaluation report with findings, implications and recommendations.
What did we find?
Over the 12-month evaluation period, around 300 people were referred into the CHIME program. Of these, almost 80% remain engaged in the program for a period that is likely to provide some treatment effect.
The findings showed that by sending CHIME staff into the hospitals to generate connections with potential clients and clinical staff, the program was able to establish effective referral pathways which is important for generating a network of support between hospital-based and community-based services.
Analysis of pre-and post- client outcome measures showed the following positive changes for clients who complete the program:
lower psychological distress
fewer unmet client needs
Improved sense of wellbeing and satisfaction with life
increased confidence and hope
greater willingness to ask for help
improved goal and success orientation
not as dominated by symptoms.
Through the evaluation, several clients’ stories were shared that demonstrated how the program leads to improved community functioning and self-management for clients, particularly in areas such as housing, social inclusion, financial independence, alcohol and drug abstinence, and setting goals.
“CHIME has helped me not be as hopeless as I was, now I’m planning for the future and I want to do more now. I’m going out again, seeing mates, I was never seeing anyone before starting this. I’m getting driving lessons, getting along better with my dad, and I want to get a job.”
CHIME client
“I was always blaming myself. [CHIME case worker] has really helped me to change my way of thinking… I’m learning to like myself… If I didn’t have that support, I wouldn’t be here today…This place is absolutely excellent, there should be more programs like CHIME.”
CHIME client
What next?
CHIME fills a gap in the local mental health service environment between acute hospital-based services and sessional services like psychological therapy and counselling. It supports people to recover from a mental illness surrounded by natural supports in their community and helps to better manage demand for acute services by preventing unnecessary hospitalisations.
The findings support the delivery of similar service models with demonstrated effectiveness achieved through partnerships with local hospital services, primary care providers and community services. Opportunities to continue to strengthen the delivery of programs such as CHIME include:
shared advisory and governance mechanisms that include consumer voice
involvement of a person’s natural supports, such as carers, family, friends, landlords etc.
providing transitional accommodation to prevent people being discharged from hospital into insecure housing or homelessness
integration with clinical supports including counselling, psychology and nursing
inclusion of peer-led practice that draws on lived experience and peer support
enhanced monitoring, evaluation and reporting of client and program outcomes in real-time
supporting the transition to the National Disability Insurance Scheme (NDIS) for those who are eligible.
Importantly, the policy environment supports the delivery of community-based and recovery-oriented mental health programs such as CHIME. For example, the Fifth National Mental Health Plan sets out a vision for a system that:
“enables recovery, prevents and detects mental illness early, and ensures that all Australians with a mental illness can access effective and appropriate treatment and community support to enable them to participate fully in the community”.
The Beacon strategies approach to evaluation
We appreciate that for most health and social service organisations, their expertise lies in delivering programs and services that generate positive outcomes for their clients, rather than the measurement and reporting of those outcomes. That’s where we come in.
We take a partnered and collaborative approach to evaluation by spending time understanding an organisation’s program and context (including program theory and program logic) and creating an evaluation plan that reflects their priorities, which may be:
monitoring the progress of implementation
understanding ‘what works’ for whom and why
identifying the extent to which objectives have been achieved
being accountable to funders through reporting
identifying potential improvements and refinements
supporting organisational or sector-wide learning
informing future decision-making and policy change.
We focus our efforts on drawing insight from data, opinion and stories to help shape evaluation reporting products that are both meaningful and useful for our client organisations.
