Getting ‘Evaluation Ready’ - the basics of program evaluation
Over the last few years, we’ve been fortunate to work with many organisations across the health and social services sectors. From small to large, government to non-government, and early-stage to very mature, each of our partner organisations shares the common goal of striving for a positive impact.
The two things we have learnt from our time spent working with these organisations are:
All of the work they do has value
They would benefit from being able to show the value of their work.
Evaluation is an important part of any program (or project, service etc.) that aims to bring about positive changes in health, social or environmental outcomes.
While specific evaluation expertise helps, too often it crowds out those people who plan and do the work. We think it’s important to challenge that and encourage more people working in these sectors to make evaluation a part of their work—no matter whether it’s their first or fiftieth evaluation.
Evaluation has evolved into one of those areas so full of technical jargon and abstract concepts, that it quickly becomes too challenging or overwhelming for the non-expert to even attempt. The definition below helps to focus on what is important in the process—drawing insight from data and doing something with it.
“ Evaluation is a type of critical thinking. It involves systematically collecting, analysing and using information to measure value, learn and make better decisions.”
adapted from Australian Evaluation Society
So what does good look like?
While every evaluation is going to use different styles and methods, we believe there is a consistent set of principles that underpin any effective evaluation.
Applying these principles will help to ensure your evaluation work is high quality, meaningful, relevant and ethical.
Evaluate across multiple domains
If you consider program ‘success’ from different perspectives, your evaluation will serve a range of purposes including accountability, learning and community engagement. Example domains include whether a program was right for the target population (appropriateness), or to what extent it met its desired objectives (effectiveness).
Use a mix of methods
Sourcing information that is both quantitative (numbers and figures) and qualitative (stories and experiences) will help to provide a richer understanding of the changes resulting from the program.
Focus on both process and impact
Understanding how a program or project was implemented can be just as insightful and meaningful as measuring the outcomes that the program generated.
Be mindful of what you claim
Consider your findings within the broader context and consider what proportion of the changes you observed can be justifiably claimed by your program.
Develop meaningful reporting products
Evaluation doesn’t have to end with a long, technical (sometimes uninspiring) written report. Communicate the insights you gather in evaluation with different audiences using mediums and messaging that meets their needs.
Commit to sharing learnings
Share your evaluation materials and reporting with colleagues across your sector to help others learn and develop from your work.
Planning an evaluation
Next time you’re planning an evaluation, think carefully about the why, what, who when and how to make sure you get the most out of your evaluation work!
Why
Understand the drivers and context for your evaluation, as this can help with decisions about what matters most. There is a range of reasons why you might choose to evaluate your program or service such as:
Tracking the progress of a program’s implementation
Measuring how much things have changed (and why)
Providing accountability to funders and sponsors of a program
Enhancing service delivery
Sharing knowledge and learning about ‘what works’ with others
Informing future decisions relating to planning, funding and policy.
What
Like planning any project, you should determine the scope based on your drivers and priorities (see above) and the resources you have available. Clear evaluation questions will help to ensure you’re evaluating what matters!
Who
Consider who should be a part of your evaluation team. The best evaluations bring in skills and knowledge from a diverse range of professional and lived experience backgrounds, such as planning design, qualitative/community engagement, ethics, cultural capability, data science/statistical analysis, critical analysis, communications, change management.
When
Evaluation activities should be taking place before, during and after your program is delivered. The best time to plan an evaluation is before the ‘doing’ has started, to make sure you’re collecting the information you need. Planning for a retrospective evaluation (i.e. looking back once everything is done) can often be challenging!
How
Developing a comprehensive evaluation plan to guide your approach will ensure you capture the best insights possible using credible data. Our approach to evaluation consists of five steps:
define your program
identify your stakeholders
create evaluation questions
collect your data
analyse and report
A note about ethics
Evaluation almost always involves people, which means there is a range of ethical issues for any evaluator to consider.
“Ethics refers to right and wrong in conduct... There is no simple recipe for ethical practice... Ethical principles rather than procedural guidelines are the final touchstone against which decisions about ethics should be made... Resolution of ethical dilemmas needs to be based on principles and will benefit from discussion and advice from professional colleagues.”
adapted from Australian Evaluation Society
Ethical issues that might arise will depend on the type of work you do and its context. Some of the more common ethical considerations include participant burden, cultural sensitivity, vulnerability, privacy, consent, reciprocal benefit and defensibility of findings.
For comprehensive references on ethical practice in evaluation, we recommend reading the AES Guidelines for the Ethical Conduct of Evaluation and Code of Ethics or IDEO’s Little Book of Design Research Ethics.
Where to next?
Evaluation Ready is a series of blog posts that aims to capture some of the knowledge, processes and lessons that we’ve assembled along our journey of supporting organisations to better evaluate their work, in the hope of prompting more people to see themselves as an evaluator.
This first post introduces some fundamental concepts around evaluation when done well. Our second post will provide a more detailed overview of the 5 step process we deploy when undertaking any program evaluation, then wrap things up with a final post about how to ‘make it stick’ within organisations and help make evaluation a routine way of working.
Read part two here