Better evaluation can boost our productivity
There’s a saying that a classic is a book that many praise but few read. Something similar could be said of policy evaluation: many praise the idea but few do it.
The OECD defines evaluation (PDF) as a systematic and analytical assessment of important aspects of a government activity with a view to creating or enhancing future performance.
Evaluation is an important part of decision making, and can help improve policies by providing feedback about what happens once the policy is put into practice. It can also help build capability among people who deliver policies and focus their attention on what matters for end users.
Evaluation can also reduce the risk of major policy failures by showing when things are starting to go off track, identify areas where the rationale for a policy no longer applies, help to compare different policies to identify which one works best and show which elements of a policy are best.
Yet New Zealand’s record in this area could be better. The Productivity Commission’s inquiry on Regulatory institutions and practices showed the costs of a weak evaluative culture. It highlighted how a “set and forget” mentality – where we only stop to review regulation when something goes badly wrong – means we miss opportunities to regularly evaluate, adjust and improve rules and ensure regulations are working as well as they should.
This can create an environment that does not tolerate mistakes, and encourages efforts to reduce the threat of criticism. Unsurprisingly, this can reduce the appetite for evaluation. If evaluation might reveal weaknesses in a policy or process and thus require change, it will not attract much support. If people benefit from an existing policy or process, they may also resist evaluation.
Even in the best circumstances evaluation can be challenging, and this resistance makes it even harder. Indeed, as SuPERU noted in their submission (PDF) to the Productivity Commission’s inquiry on More effective social services, the ability of evaluation to improve efficiency and effectiveness also tends to be held back by:
- not enough consideration of research, evaluation and monitoring when policy is being designed and implemented. What performance indicators will be used and how data will be collected need to be considered when a policy is being developed, not just when its impact is being evaluated at a later date;
- a focus on evaluating the success of individual programmes, when a system-wide approach would improve understanding; and
- unclear outcomes and short term funding streams, which encourage a focus on inputs and outputs rather than long-term outcomes.
However, it isn’t all bad news. There are plenty of examples of evaluation in New Zealand which are well planned and robust. As SuPERU has shown, in the social services sector these include:
- the Early Start Programme commissioned by the Ministry of Social Development;
- Auckland City Mission’s “Family 100 Research Project”; and
- the cross-agency study of the Incredible Years Parenting courses.
And looking forward, the appetite for evaluation seems to be strengthening too. Just a few weeks ago the Productivity Commission was one of 24 public sector agencies celebrating the official launch of the International Year of Evaluation. This year includes a programme of events chosen specifically to develop and embed an evaluation culture in the public sector. The hope is that, by the end of the year, we can expect many more public service leaders to look to evaluation to identify areas for improvement and to help re-prioritise effort.
- Patrick Nolan, Principal Advisor
Disclaimer: Blog posts are written by staff members and do not represent the official views of the Productivity Commission.
Photo courtesy of MBIE.