Change Detected: What to Do Next
Bootstrap CUSUM has found a statistically significant shift in your data — a change point. Before celebrating or reacting, three questions need honest answers: which direction did it move, does the date make sense, and is it sustained? The checklist below takes you through each one.
If the StepChange Analyzer (Bootstrap CUSUM) detects a change point, investigate first: confirm direction, verify the date, rule out artefacts (changes in how the data was collected or defined), and check balancing measures to confirm the system improved rather than shifting harm elsewhere.
A change point tells you that the system shifted and roughly when — not why. Attribution is your job. Choose the response level (Joiner 1/2/3) and re-test with Bootstrap CUSUM.
If Bootstrap CUSUM (StepChange Analyzer) detects a change point, the default action is to investigate — not to celebrate or react. Confirm the direction, verify the date against known events, rule out artefacts, and check balancing measures before attributing the change to any intervention.
A change point tells you that the system shifted and approximately when. It does not tell you why. Attribution is your job. Apply Joiner's levels of fix to determine whether the response should be at Level 1, 2, or 3 — then re-test with Bootstrap CUSUM to confirm the response worked.
☰ Contents
What “change detected” means
A change point is a statistically significant, sustained shift from one stable level to another. The Bootstrap CUSUM algorithm has determined — at your chosen confidence threshold — that the data did not merely fluctuate randomly: something structural changed. The shift is real enough that it would be unlikely to appear by chance alone in a system that had stayed the same.
What it does not mean: the change was caused by what you think it was caused by. The algorithm identifies that a shift occurred and approximately when. It says nothing about why. Attribution is your job, not the algorithm’s — and it is the step most often skipped.
A confirmed change point is the data telling you something real happened. The investigation starts with a date — and works backwards to ask what changed in the system at or just before that date. The checklist below is that investigation.
First checks — before celebrating or reacting
🔎 Three questions to answer before acting
Change point investigation checklist
Work through each item below for the change point date the algorithm returned. The goal is to distinguish a genuine structural shift in the system from four common false signals.
| Item | What to look for | Signal if present |
|---|---|---|
| 1. What changed in the system at or just before the change point date? | List every intervention, policy change, leadership change, capital investment, or operational change that was implemented in the 1–3 months before the change point date. One of these is the candidate cause. If none can be identified, the change point needs more investigation before attribution. | If a credible intervention matches the date: genuine structural change is plausible. If no intervention matches: investigate artefact causes below. |
| 2. Definition or coding change? | Did the definition of what is being measured change around the change point date? Common examples: a new diagnostic code introduced, a reporting threshold revised, a data field redefined, a new IT system replacing an old one, a change in who submits the data. A definition change produces a change point in the data that has nothing to do with the underlying system. | If a definition change coincides with the date: the change point is a measurement artefact. Rerun the analysis from the point where the new definition stabilised. |
| 3. Denominator shift? | If the metric is a rate (events per 1,000 admissions, per 100 procedures, per population), check whether the denominator changed. A fall in the denominator without a fall in the numerator produces an apparent rate increase; a rise in the denominator produces an apparent rate decrease. Both are artefacts. | If the denominator shifted: replot numerator and denominator separately and check whether the numerator independently changed at the same point. |
| 4. Case-mix or population change? | Did the composition of the population being measured change? For NHS data: a change in referral patterns, a new pathway diverting a specific patient group, a demographic shift in the catchment. A case-mix change can produce a genuine change in the metric without any change in the process being measured. | If case-mix shifted: stratify the data by patient group and rerun. A change point that disappears on stratification was a case-mix artefact. |
| 5. Seasonal pattern or one-off event? | Check whether the change point coincides with a predictable seasonal transition (summer vs winter, academic year start, financial year) or a one-off event (industrial action, pandemic, severe weather). Bootstrap CUSUM is designed to detect structural shifts, not seasonal cycles — but a strong seasonal signal in limited data can sometimes produce a spurious change point at the seasonal boundary. | If seasonal: run the analysis across a longer window spanning at least two full seasonal cycles. A genuine structural change point will be visible across both cycles; a seasonal artefact will not. |
A change point in air quality data was attributed to a congestion or emissions zone policy. The change point date returned by the algorithm preceded the policy implementation date by several weeks. The actual cause was a change in monitoring station calibration protocol — a definition change — that happened to coincide with the policy debate. The policy was credited with an improvement it had not yet had time to cause. Attribution errors of this type are not rare: they are the default when investigators start from a desired conclusion rather than from the change point date. Start from the date. Work backwards.
What to do next
📈 Two paths from a confirmed change point
Joiner’s rule (p.138, Fourth Generation Management) requires this question before any other response. A change point in the improvement direction could be either: a special cause — a one-off event, a new staff member, a temporary resource injection, a data artefact — or a genuine structural system improvement. The response to each is different, and confusing them is the most common error after a positive change point.
A special cause improvement should be identified, understood, and if possible standardised — but it does not mean the system has changed. Remove the special cause and the metric will drift back. A structural improvement means the system itself has changed; the new level should be self-sustaining once the conditions are standardised.
Use the special cause variation checklist to determine which you are looking at: does the change point date correspond to a specific identifiable event (special cause), or to a redesign of the system’s processes or structure (system improvement)? The investigation checklist above is the tool for this determination.
Then: hold steady, avoid tampering, and confirm across the next hard season.
Once the type of change is established, the correct response is to standardise the conditions that produced the shift and hold them stable while the new level is confirmed. The most common mistake is to intensify the intervention — Deming called this tampering: adding changes to a system that has already stabilised at a new level, introducing unnecessary variation.
Document exactly what changed and when, standardise the new process so it cannot drift back, and monitor through the next hard season before claiming the change is permanent. A genuine structural improvement will survive winter (or its equivalent). A temporary special cause improvement will not — and Bootstrap CUSUM will detect the reversal.
Also check your balancing measures. An improvement in one metric achieved by worsening another is not system improvement. It is redistribution.
A deterioration change point means the system has structurally shifted to a worse level. The immediate priority is to stop the deterioration from deepening — triage and stabilise. This is Joiner’s Level 1 (fix the output) and is appropriate as an emergency response, not a permanent solution.
Once stabilised, the investigation turns to why the system shifted. Apply the investigation checklist above. Then apply Joiner’s levels: is the cause addressable within existing process boundaries (Level 2), or does it require a structural change that crosses organisational or system boundaries (Level 3)? A Level 2 response to a Level 3 cause will stabilise the metric temporarily and then see it deteriorate again — which Bootstrap CUSUM will detect as a flat line followed by a further downward change point.
See Theory of Constraints for the framework for identifying the binding constraint before intervening.
📝 The Deming question — “By what method?”
Before claiming that a programme, policy, or intervention caused the change point, answer Deming’s question precisely: by what method did it produce the effect? Describe the mechanism — the chain of cause and effect — that connects the intervention to the change point date and the magnitude of the shift. If the mechanism cannot be described, the attribution is a hypothesis, not a conclusion. The mechanism description is also your prediction for what will happen if the conditions are reversed — which is the test of whether you understand the cause or merely coincided with it.
Run it again — recommended re-tests
A single Bootstrap CUSUM run is a starting point. The following re-tests help confirm whether the change point is robust or sensitive to analytical choices.
| Re-test | Why | What to look for |
|---|---|---|
| Different confidence threshold | A change point detected at 95% confidence but not at 99% is a weaker signal than one that survives at 99%. A change point that only appears at 80% may be marginal. Test at 90%, 95%, and 99%. | A robust change point survives across all three thresholds and returns approximately the same date. A fragile change point disappears or moves significantly as confidence rises. |
| Extend the window | Adding more post-change-point data strengthens or weakens the signal. If the change point was detected close to the end of your data series, it may not yet have enough evidence behind it to be confirmed structural. | Re-run as new data arrives each month. A genuine structural shift will remain visible and its date will be stable. A borderline detection may disappear as more data reveals the shift was temporary. |
| Check balancing measures | Run Bootstrap CUSUM on the balancing measures identified for this metric. An improvement change point in the primary metric should not coincide with a deterioration change point in the balancing measure. | If primary improves and balancing worsens at the same date: the problem has moved, not been solved. If both improve or both hold flat: the primary improvement is more credible. |
| Stratify by subgroup | If the metric is an aggregate, run Bootstrap CUSUM on the subgroup data separately (by site, by patient group, by season). A genuine structural change will usually be visible across most subgroups. A change point driven by one outlier subgroup is a different — and more limited — finding. | Consistent change points across subgroups strengthen the aggregate finding. A change point in one subgroup only suggests the effect is local and the aggregate change point may be misleading. |
| Check the leading indicator | If a lead indicator exists for this metric, run Bootstrap CUSUM on it too. A genuine structural shift in an outcome measure should be preceded by a change point in the leading indicator — with a lag consistent with the known mechanism. | Lead indicator change point before outcome change point, with the right lag: the causal chain is intact. Outcome change point without lead indicator change point: investigate alternative causes or artefacts. |
(1) Direction confirmed → (2) Date investigated against known events → (3) Artefact causes ruled out → (4) Mechanism described → (5) Balancing measures checked → (6) Confirmed across next hard season. Only after step 6 is the change point a defensible claim of sustained structural change. Steps 1–5 are the honest work between detection and conclusion.
Re-run the analysis
Upload your data with a longer window, a different confidence threshold, or your balancing measure series — and run Bootstrap CUSUM again to test the robustness of the change point.
▶ Open the StepChange Analyzer