Introduction
The evidence decision-making tool assists you to:
- assess how confident you are that a certain policy, program or other initiative is likely to be effective in your context
- decide on next steps, including how to implement the initiative given your level of confidence and how to collect more evidence to increase your confidence in its effectiveness.
The tool can be used by an individual or a group – for example, in a planning workshop. It’s designed to be flexible, so you can use it to consider a change to an existing initiative or the introduction of something new.
The tool provides a structured way to help you consider the rigour and relevance of the evidence supporting an initiative objectively, but it’s not a set of rigid rules to follow. Sometimes, it will be difficult to decide which category your evidence fits into and you’ll need to use your professional judgement to make a decision.
There is also an Evidence Decision-Making Tool for Educators and Teachers in early childhood education and care or school settings to consider practice or program changes.
Using the evidence decision-making tool
Read the descriptions of evidence in the first section of the tool and identify the statement(s) that best describes the evidence you currently have to show that this initiative is likely to be effective in your context. The highest level you reach is your current level of confidence.
For example, if you have heard anecdotal evidence (which is classified as low confidence evidence) and you have also read peer-reviewed research that suggests high-performing systems use this initiative (which is medium confidence evidence), then your current confidence is at level 2, medium confidence.
Read the suggestions about implementing the approach in the corresponding section under 'how should I proceed'. These suggestions include how to collect more evidence if you seek to increase your confidence in the effectiveness of the initiative. For example, if you identified that your level of confidence is low in step 1, you would consider the suggestions in the low confidence section.
You can increase your confidence by finding more rigorous and/or relevant research evidence or by trialling the initiative and observing or collecting data about the effects. When your confidence increases, move up a level in the tool and consider the suggestions that match your new level of confidence.
Deciding whether you have enough confidence
Although high or very high confidence is preferable, sometimes there isn't enough evidence available to achieve these standards.
When evidence is lacking, use your professional judgement to weigh the potential benefits, costs and risks of implementing the initiative. For example, it may be acceptable to implement an initiative in which you have only low or medium confidence when not changing may do harm or when the consequences of implementing an ineffective initiative are small or have been appropriately mitigated. In these instances, it's particularly important to carefully monitor and evaluate the effects of the initiative. The evidence decision-making tool provides guidance for cautiously implementing an initiative and collecting more evidence when you have only low or medium confidence.
Remember, knowledge is constantly evolving and contexts change so the process of considering the evidence is never finished. It’s a good idea to revisit the evidence and reassess your confidence level as often as is reasonable given the importance of the decision you’re making.
Evidence decision-making tool for policymakers
Determining confidence in the effectiveness of an initiative in your context
Identify the statement(s) that best describes the evidence you currently have to support this initiative. Your current level of confidence is the highest level for which you have evidence. For example, if you have both level 1 and level 3 evidence, your current level of confidence is level 3.
I can hypothesise why the initiative will work. I can explain step-by-step how the initiative will lead to positive effects in my system. However, I don’t have any data or academic research to show that this hypothesis is correct.
and/or
I have read descriptions of the initiative that provide opinion or anecdotal evidence. However, these descriptions don't provide data or references to academic research that show the initiative is associated with positive effects.
Early adopters in my system think the initiative is effective, and can show me data that suggest there have been positive effects.
and/or
I have read research published in peer-reviewed academic journals or conducted by a trustworthy source such as AERO that suggests high-performing systems use this initiative. This evidence shows that using this initiative is associated with positive effects but doesn’t demonstrate that the initiative caused the positive effects.
I have read research published in peer-reviewed academic journals or conducted by trustworthy sources such as AERO that evaluates this initiative. This research shows this initiative causes positive effects; however, the research was not necessarily in my context.
and/or
I have read credible nonacademic sources (for example, books, reports) that said this initiative has positive effects. These sources included references to peer-reviewed academic research or research conducted by a trustworthy source such as AERO that show the initiative causes positive effects.
I have read systematic reviews and/or metaanalyses that show this initiative causes positive effects over a wide range of contexts.
and/or
I have read or conducted a rigorous causal evaluation of the initiative as implemented
in my system. This evaluation shows that the initiative causes positive effects.
and/or
I have read several research studies published in peer-reviewed academic journals or conducted by a trustworthy source such as AERO that show this initiative causes positive effects in system contexts similar to mine. These studies used robust experimental designs.
Proceeding based on your level of confidence
Read the suggestions in the section that matches your current level of confidence.
I should implement the initiative cautiously or not at all.
If I decide to trial it, I should:
- run a small-scale pilot led by experienced leaders, teachers and/ or educators who want to trial the initiative
- undertake a risk assessment to plan for and mitigate any consequences should the initiative be ineffective
- closely monitor implementation to identify whether the initiative is working and manage any unintended negative consequences
- be prepared to discontinue the initiative if it appears no more effective than existing approaches or if there are signs of negative consequences.
To increase my confidence, I could:
- undertake a review of the research literature to identify whether there is any evidence from other systems that provides medium, high or very high levels of confidence about whether the initiative is effective
- consult others who have implemented the initiative to learn if they have data or research evidence that demonstrates the initiative has positive effects.
I should implement the initiative cautiously.
When implementing the initiative I should:
- allocate limited resources to the initiative and ensure that I don't divert resources from other initiatives that I have more confidence in
- replicate what has been done in other systems where the initiative is associated with positive effects
- undertake a risk assessment to plan for and mitigate any consequences should the initiative be ineffective
- closely monitor implementation to identify whether the initiative is working and manage any unintended negative consequences
- be aware that the initiative may be ineffective or have negative consequences and be prepared to discontinue it if the evidence doesn’t support its use.
To increase my confidence, I could:
- undertake a review of the research literature to identify and synthesise evidence that provides high or very high levels of confidence that the initiative is effective
- commission an evaluation of the initiative (which could be of a small-scale pilot) to assess whether it causes positive effects.
I should implement the initiative.
When implementing the initiative I should:
- ensure that I don't divert resources from other initiatives that I have more confidence in
- replicate what has been done in other contexts where the initiative has demonstrated positive effects
- begin implementation in places that are most similar to those where the initiative was found to be effective
- carefully consider whether adjustments are necessary for my system, base them on insights from the research about how the initiative produces positive effects, and document the adjustments and why they were made
- undertake a risk assessment to plan for and mitigate any unintended consequences
- monitor implementation to identify whether the initiative is working and manage any unintended negative consequences.
To increase my confidence, I could conduct or commission research or evaluation to assess the effectiveness of the initiative in my system – for example, by:
- comparing the longitudinal outcomes of schools or services that have already implemented the initiative with similar schools or services that haven't
- randomly staggering implementation and comparing the outcomes of schools or services that start implementation at different times.
I should implement the initiative as widely as possible.
When implementing the initiative I should:
- allocate resources that support implementation of the initiative at scale
- replicate how the initiative was implemented in the research where it demonstrated positive effects
- collect and analyse data on how and why the initiative works in my system to support ongoing improvement
- continually monitor the initiative to ensure ongoing effectiveness.
To increase my confidence, I could conduct or commission research or evaluation to
better understand:
- why the initiative is effective, its key features and any necessary enabling conditions
- whether it is equally effective for all children, students and/or educators, and if not, why this is the case.
Remember to keep monitoring the research evidence to stay up-to-date with evolving knowledge about effective practices, programs and policies in education.
Keywords: practice implementation