No change detected: decide to wait (don’t tamper)
Bootstrap CUSUM found no structural change. The most important thing you can do now is nothing — deliberately, defensibly, with a review date. This page gives you the decision rules, the evidence threshold, and the language to hold the line when pressure mounts to act.
If there is no signal, default to waiting and measuring — do not turn the dials just to show activity.
This is not passive. It is the active decision that prevents the most common and most damaging error in improvement work: intervening again before the first intervention has had time to work. Deming called this tampering. Bootstrap CUSUM is the tool that tells you when you are in a tampering situation — and gives you the statistical evidence to defend the decision to wait.
If no change is detected, default to waiting (don’t tamper): set a review date, improve the theory/measure/test, and re-run after enough new data points.
Launching a new initiative to show activity before the signal has resolved adds variation without improving the system. Set a pre-committed review date and re-run Bootstrap CUSUM with additional data.
☰ Contents
Why waiting is the correct default
Every system produces variation. Some months are better than average. Some are worse. If you intervene every time a result is below your target or below the previous month, you are not responding to genuine deterioration — you are responding to noise. And each intervention adds new variation to the system, making the next period’s results harder to interpret.
Deming called this tampering — one of the most common and most damaging management behaviours in any organisation. The cost is real: wasted resource on interventions that were not needed, demoralised staff who are constantly told to change what they are doing, and a system whose underlying performance becomes impossible to read because it is always in the middle of another initiative.
Deming illustrated tampering with his funnel experiment. A marble dropped through a funnel lands near a target on a table. Common variation means it never lands in exactly the same spot twice. If you adjust the funnel position after each drop to correct for the last error, the marbles scatter further and further from the target. The adjustment — well-intentioned, apparently rational — creates more variation, not less. The correct response to common cause variation is to do nothing and let the system settle. The correct response to a genuine signal is to investigate and change the system. Bootstrap CUSUM distinguishes the two.
The flat Bootstrap CUSUM line is not bad news. It is honest information. It tells you that the variation you are seeing is common cause variation — the normal noise of a system operating at its current level. The underlying process has not changed. The correct response is to keep measuring, maintain your current intervention, and wait for the statistical evidence that the system has moved to a new level. If that evidence does not appear within your review window, you have learned something important: the intervention was not at the right level. That is the prompt to return to root cause — not to add more interventions on top of the one that is not yet confirmed as working.
The decision rules — when to wait, when to act
| Situation | Bootstrap CUSUM says | Correct action | Wrong action |
|---|---|---|---|
| Result is below target but within normal variation | No change point. All points within natural process limits on X-mR chart. | Wait. The system is operating normally. The target may be wrong, but the system has not changed. | Launch a new initiative in response to a single bad month. This is tampering. |
| Result is worse than last month | No change point. Month-to-month variation is common cause. | Wait. Last month being better than this month is expected. It does not mean the system has deteriorated. | Hold a review meeting to identify what went wrong. Nothing went wrong that wasn't already happening. |
| Intervention was implemented 3 months ago — no change yet | No change point. Insufficient post-intervention data. | Wait. Most interventions require 6–18 months before a structural change is detectable. Set a review date and monitor. | Conclude the intervention failed and launch a replacement. The original may still be working. |
| Board is asking why results haven’t improved after 6 months | No change point at 95% confidence. Possible movement at lower confidence. | Report honestly. The Bootstrap CUSUM has not yet confirmed structural change. The pre-committed review date is [month]. You will report again then with the statistical evidence. | Cherry-pick a favourable comparison period to show improvement. This undermines the credibility of every future result. |
| A single result breaches a threshold (outlier) | Single point outside natural process limits on X-mR. No sustained change point on Bootstrap CUSUM. | Investigate the single event. Find out what specifically happened in that period. This is a special cause — it has a specific, identifiable cause. Find it. | Change the whole system in response to one outlier. The system did not change — one period was unusual. |
| Bootstrap CUSUM shows a change point — downward (harm) | Confirmed deterioration change point at 95% or above. | Act. The system has structurally deteriorated. This is not variation. Return to Step 1 of the improvement method immediately. | Wait. This is the one situation where waiting is wrong. |
The Bootstrap CUSUM evidence threshold
The confidence level you set in Bootstrap CUSUM is your evidence threshold — the statistical bar that a change must clear before you declare it structural. Choosing the right threshold for your context is as important as choosing the right outcome measure.
Use for early warning systems where the cost of missing a signal is high. Accept more false alarms in exchange for earlier detection. Appropriate for safety-critical metrics where early action prevents harm. A 90% change point is a signal to investigate, not to declare success.
The appropriate threshold for most improvement programme evaluation. A 95% change point means that if the process had not genuinely changed, you would see a false alarm of this size only 1 in 20 times. This is the threshold recommended for most NHS improvement reporting.
Use before committing to irreversible action: scaling an intervention nationally, decommissioning a service, or making a major capital investment on the basis of improved performance. A 99.7% change point is rare false-alarm territory — you can be confident the system has genuinely moved.
Wait until Bootstrap CUSUM reaches your pre-committed confidence threshold — or until your review date, whichever comes first. At the review date, one of three things will be true:
1. Change point confirmed at your threshold. The intervention worked. Investigate the mechanism. Document it. Keep monitoring.
2. No change point, but approaching your threshold. The system may be moving slowly. Extend the review date by 6 months and monitor. Do not add further interventions.
3. No change point, no movement. The intervention did not produce structural change within the expected window. Return to root cause analysis. The constraint was not addressed at the right level. See Joiner’s Levels of Fix.
How to document the decision to wait
The decision to wait is only defensible if it is documented before the pressure arrives. A verbal commitment to wait is not a commitment — it is an intention, and intentions bend under board pressure, political scrutiny, or the anxiety of a run of poor results. Write it down before the intervention is implemented.
📝 The pre-committed waiting declaration
Write this before implementation and share it with your governance group:
Intervention: [Name the specific intervention]
Implemented: [Date]
Outcome measure: [The specific metric Bootstrap CUSUM will test]
Expected direction: [Up or down]
Confidence threshold: [90% / 95% / 99.7%]
Review date: [Date — typically 12–18 months after implementation]
Pre-committed rule: If Bootstrap CUSUM has not confirmed a change point at [threshold]% by [review date], we will return to Step 3 (root cause) rather than adding further interventions.
This document does three things. It creates a pre-specified prediction that cannot be reverse-engineered after the results arrive. It gives you the language to resist pressure to intervene before the review date. And it records the hypothesis so that a future Bootstrap CUSUM change point — or absence of one — can be interpreted correctly.
When pressure mounts — how to defend the decision
The hardest moment is not choosing to wait. It is holding the decision when results look bad, when the board wants answers, and when colleagues are suggesting new initiatives. These are the responses that work.
“Results are worse than last month — we need to do something”
Response: “Bootstrap CUSUM on the full series shows no structural change. Month-to-month variation of this size is expected. Adding an intervention now would be responding to noise, not to a genuine deterioration. Our review date is [date]. If the Bootstrap CUSUM shows a confirmed downward change point before then, we will act immediately.”
“We have been waiting 6 months and nothing has changed”
Response: “That is correct — Bootstrap CUSUM has not yet confirmed a structural change point. Six months is within the expected lag for this type of intervention. Our pre-committed review date is [date]. If there is no change point by then, we will return to root cause analysis. Adding further interventions now would make it impossible to attribute any future change point to the original intervention.”
“Other organisations are doing [new initiative] and getting results”
Response: “We would want to see the Bootstrap CUSUM analysis on their data before changing our approach. Before-and-after comparisons almost always show improvement — the question is whether the improvement is a genuine structural change point or seasonal variation in a short window. If their data shows a confirmed change point at 95% confidence sustained across two periods, that is evidence worth acting on. Otherwise we risk abandoning an intervention that may still be working.”
“The data shows improvement if we look at it this way”
Response: “The pre-committed test was Bootstrap CUSUM on [specific measure] at [confidence threshold]%. That test has not confirmed structural change. Choosing a different analysis after seeing the data is the measurement equivalent of moving the goalposts. Our credibility depends on the pre-committed test being the one we report, regardless of what it shows.”
When waiting is wrong — the three exceptions
Deciding to wait is the correct default when Bootstrap CUSUM finds no structural change. There are three situations where it is not correct.
A confirmed downward change point in a harm measure, or upward change point in a waiting time or error rate, is evidence that the system has structurally deteriorated. This is not variation. Act immediately. Return to Step 1 of the improvement method. Do not wait for the review date.
An X-mR chart signal — a single point outside the natural process limits — in a patient safety or clinical harm measure requires investigation even if Bootstrap CUSUM has not confirmed a structural change. The signal may be a one-off special cause event. Find its specific cause. This is different from tampering — you are investigating a specific event, not changing the system in response to variation.
If the pre-committed review date has passed and Bootstrap CUSUM has not confirmed a structural change at your threshold, the correct action is to return to root cause analysis — not to keep waiting indefinitely. The flat line is information: the intervention was not at the right level, or did not address the dominant constraint. See Joiner’s Levels of Fix and Theory of Constraints for the diagnostic framework.
Setting the review date
The review date is not arbitrary. It should be set based on the expected lag between the intervention and a detectable change in the outcome measure. Different types of intervention produce change point signals at different speeds.
| Intervention type | Expected lag | Recommended review date | NHS example |
|---|---|---|---|
| Engineering control / structural redesign | 1–6 months | 6 months post-implementation | NRFit connector mandate, ULEZ charging zone. If the system changed, the change point appears quickly. |
| Process change / pathway redesign | 3–12 months | 12 months post-implementation | New discharge pathway, revised sepsis bundle. Process changes take time to become routine across all shifts and staff. |
| Workforce or training intervention | 6–18 months | 18 months post-implementation | GP dementia awareness training. Training effects accumulate over referral cohorts and are slow to appear in outcome data. |
| Cultural or leadership change | 12–36 months | 24–36 months post-implementation | Rejection of “culture of accepting” (Watford General). Culture changes are slow to stabilise and require sustained leadership to hold. |
| System capacity investment | 12–36 months | 24 months post-investment | New intermediate care beds. Capacity investments require commissioning, staffing, and patient pathway changes before the outcome measure moves. |
| Prevention programme | 5–20 years | Annual review with 5-year horizon | Falls prevention, cardiac screening. Prevention effects are visible only when the cohort of prevented events is large enough to move the aggregate series. |
Most NHS improvement programmes operate on a political or financial cycle of 1–4 years. Most structural improvement interventions require 12–36 months before Bootstrap CUSUM can confirm a genuine change point. The mismatch between the political cycle and the improvement lag is one of the most reliable predictors of a flat Bootstrap CUSUM line — not because the intervention failed, but because it was abandoned or replaced before the evidence could appear. Pre-committing the review date in writing — and sharing it with the governance body — is the only practical protection against this trap.
Run Bootstrap CUSUM on your outcome measure
Upload your CSV to the StepChange Analyzer. The result will tell you whether to wait or act — and give you the confidence level and change point date to back the decision.
▶ Open the StepChange Analyzer