Timing is everything: Primary Health Networks (PHNs) and program evaluation
All Primary Health Networks (PHNs) operate under a similar framework that outlines a systematic approach to how they commission services - that is a cycle of needs assessment, co-design, implementation and evaluation. Many PHNs are maturing in their approach to commissioning services and are starting to get a feel for their role as a commissioner. Central to being an effective commissioner of health services is the ability to use insight to make better investment decisions.
Given that the Beacon Strategies team has been involved in a number of evaluation and program reviews in the PHN environment we think it is timely to share a few tips in relation to commissioning effective evaluations and program reviews.
Undoubtedly in our eyes, the biggest issue when commissioning evaluations and program reviews is managing timeframes. PHNs will often acknowledge that they need to evaluate or review a program prior to any recommissioning or redesign work. In some cases, an evaluation will be initiated but decisions are made about the future of the program prior to the conclusion of the evaluation.
How can PHNs ensure meaningful and timely evaluation and program reviews?
PHNs can tackle this through committing to two courses of action 1) improving the timing of evaluations and 2) adjusting their expectations and level of rigour. Here's how.
Improving the timing of evaluations
PHNs need to work back from the end date of a commissioned program. By factoring in the time it would take to consider evaluation findings then conduct commissioning activities, PHNs can schedule the program evaluation and/or review with enough time for a meaningful decision to be made by portfolio leads, commissioning staff and senior leaders. This may seem like an over simplistic solution but it is fundamental in ensuring the timely delivery of information prior to making decisions. We’ve broken down an example timeframe for a hypothetical 2 year project below:
Service delivery commences (month 1)
Evaluation commences (month 18)
Evaluation concludes (month 20)
Sense making and internal decision processes (month 21)
Commissioning process (month 22 - 24)
The timeline above can present a number of challenges for PHNs, particularly taking into consideration the short timeframe of contracts issued to service providers. Nonetheless, timing an evaluation activity or review point is critical in making informed commissioning decisions.
Communication and change management processes will be far more effective to implement with appropriate lead in time to manage the expectations of current commissioned services. Being transparent as early as possible with current providers and internal staff regarding future changes and their role in the change will go along way to maintaining the positive and productive relationships PHN have internally and with their sector partners.
Adjusting expectations
PHNs may also need to adjust their expectations in relation to the robustness and design of program evaluations and reviews. Evaluation for this purpose is all too often a largely academic process of deeming a program a ‘success’ or ‘failure’ relying on a small number of indicators. This understanding of ‘evaluation’ is flawed in the context of PHNs due to short contract timeframes and the need for timely information about a program to make prompt commissioning decisions. Yes, evaluation and program reviews need to be robust and defensible but they also need to be delivered in a timely fashion.
For PHNs and evaluators, this means acknowledging that you may not have the complete picture prior to making a funding decision - an issue this is particularly relevant when trying to quantify the impact or cost-effectiveness of a program. For evaluators, there needs to be a dedicated focus on process evaluation and monitoring of short term indicators to assess whether a program or service is on track to deliver its outcomes.