What to Do Next
You have run Bootstrap CUSUM. Pick your result below. Each pathway gives you the next 3–5 actions — not theory, just what to do.
Visit Interpret results first — common pitfalls, seasonality, multiple change points, and the difference between signal and noise.
Change detected
Bootstrap CUSUM has confirmed a structural shift. Something in the system changed. Now do these five things.
-
1Check the direction. Is this an improvement or a deterioration? An upward change point in harm events is a warning. A downward change point in waiting times is success. Read the direction before drawing any conclusions.
-
2Investigate what changed in the weeks before the change point date. The CUSUM dates the confirmation — the actual cause predates it by weeks or months. List everything that changed in the system around that time: staffing, process, policy, external events. Use 5 Whys to trace the cause.
-
3Confirm the change is sustained. Re-run Bootstrap CUSUM on the full series including recent data. A temporary excursion that reverts is not structural improvement. You want the new level to be holding.
-
4Document the mechanism, not just the correlation. Why did this work? Was it a Level 3 fix? Did it address the dominant constraint? Write it down — that is what makes the learning transferable to other settings.
-
5Keep monitoring. Run Bootstrap CUSUM on the same measure every 6 months. A new change point in the wrong direction is your early warning of regression. The earlier you catch it, the more options you have.
Return to Step 1 of the 7-step method. List symptoms, find root cause, identify constraint, design a structural fix. Do not launch multiple simultaneous responses — that is tampering.
No change detected
If there is no signal, default to waiting and measuring — do not turn the dials just to show activity.
This is not passive. It is the active decision that prevents tampering. Document it formally so it is defensible:
- What intervention was made and when
- What outcome measure you are tracking
- Your review date — typically 6–12 months after implementation
- Your pre-committed rule: "If no change point by [date], return to Step 3"
Without a documented review date, "wait" becomes indefinite delay. With one, it is a disciplined hypothesis.
If the review date has passed and there is still no signal, choose one of these:
-
1Return to root cause. Go back to Step 3 of the 7-step method. Did the analysis stop at "human error"? Was the Joiner test applied? Was the fix at Level 1 or 2 rather than Level 3? A flat CUSUM after genuine implementation is almost always a root cause problem, not an implementation problem.
-
2Check the measure. Are you tracking an outcome metric or a process metric? A process can improve while the outcome stays flat. Ask: does this measure capture what the system is actually for? See Types of measures.
-
3Check the theory. Did the intervention address all necessary conditions, or just one? See Necessary But Not Sufficient. Did it address the dominant constraint, or something upstream of it? See Theory of Constraints.
A flat CUSUM line is honest information most measurement systems cannot provide. Standard before-and-after comparisons almost always find improvement — because the window is chosen to show one. Bootstrap CUSUM is indifferent to the narrative. Organisations that use it consistently stop receiving credit for changes that did not occur — and start understanding what actually moves their systems. See also: False Alarms in Performance Charts.
Data problem — fix and re-run
The result is unreliable because of a problem with the data series. The table below maps the most common problems to their fixes.
| Problem | Fix |
|---|---|
| Fewer than 20 data points | Extend the historical series. If not possible, reduce frequency — quarterly instead of monthly gives more historical points. Annual data removes seasonal variation and extends effective history further. |
| Missing values (gaps) | For 1–2 gaps, interpolate using the average of surrounding values. For longer gaps, split the series and run Bootstrap CUSUM on each segment separately. Never fill gaps with zeros unless zero is a genuine observed value. |
| Date format not recognised | Use ISO format: YYYY-MM-DD (e.g. 2024-03-01). For monthly data, use the first day of each month. Avoid DD/MM/YYYY — ambiguous between UK and US conventions. |
| Mixed frequencies | Standardise to a single frequency before running. Aggregate everything to quarterly, or split the series at the point where frequency changes. |
| Extreme outlier distorting the baseline | Identify the cause. If it is a genuine one-off (e.g. COVID lockdown), run Bootstrap CUSUM excluding that period and note the exclusion explicitly. Never remove outliers without documenting why. |
| Too soon after an intervention | Less than 6 months of post-intervention data is not enough. Set a review date and re-run then. Do not interpret "too soon to tell" as "no change." |
| Denominator changed mid-series | If it is a rate and the denominator changed (e.g. patient population grew), recalculate all values using a consistent denominator, or split the series at the point of change. |
-
1Fix the specific problem above. Make only the minimum change needed. Document every change to the data and why.
-
2Re-run Bootstrap CUSUM. Change point found → go to Path A. No change → go to Path B.
-
3If problems keep appearing, reconsider the measure itself. The data quality issue may be a symptom that the measure was never properly defined. See Types of measures. If the series has strong seasonal patterns, see Seasonality Mistaken for Improvement before re-running.