🔎 Improvement Diagnostic
Why Improvement Programmes Fail
Every improvement programme that has not produced a Bootstrap CUSUM change point has failed for one of a small number of reasons. Find the question that describes your situation. Follow the link to the page that answers it.
How to use this page
Read the questions in each cluster. Find the one that best describes what you are experiencing. The link takes you to the page that diagnoses the cause and provides the instrument for making the invisible visible. You do not need to read the whole site. You need to find your question.
If you are not sure which question fits: Start with the six-step Visibility Loop →
1
Not seeing the problem clearly
Visibility Loop Steps 1 & 2
Step 1 — See the pattern
We can’t see the wood for the trees — we’re too close to it
When everyone is inside the system, the pattern it produces is invisible. The event is visible. The rate that produces events routinely is not. Bootstrap CUSUM on the event series makes the pattern visible and dates when it started.
Step 1 — See the pattern
We don’t appreciate that systems thinking applies here
Seeing the system rather than the individual event is the prerequisite for everything else. Without it, every event looks like an individual failure rather than a system output.
Step 2 — Find the structure
We haven’t got to root causes — we’re stopping too early
Five Whys applied in full reaches the structural cause. Most RCA stops at Why 2 or Why 3 — at the process or the person — and produces a Level 1 or Level 2 fix. The structural cause remains untouched.
Step 2 — Find the structure
We’re not going deep enough — the investigation isn’t reaching the cause
The investigation authority stops at the ward, the department, or the team boundary. The structural cause sits in procurement, IT, or a different budget — outside that boundary. The Boundary Trap.
Step 3 — Name the belief
We’re not understanding the type of problem or improvement needed
Some problems are common cause variation in a stable system. No amount of investigation or intervention will change them — only a structural redesign will. Deming called reacting to common cause as if it were special cause “tampering.”
Step 3 — Name the belief
We’re not appreciating constraints — we don’t know where the bottleneck is
The constraint determines the throughput of the entire system. Improving anything other than the constraint produces no improvement in output. The pile of work is visible — the station limiting flow is not.
2
Not knowing whether it has worked
Visibility Loop Step 5
Step 5 — Test honestly
We don’t know whether our intervention has actually worked
Action completion is not improvement. A change point in the outcome series is improvement. Bootstrap CUSUM with a pre-committed prediction is the honest test. Without it, any result can be rationalised as success after the fact.
Step 5 — Test honestly
We’re not measuring the right things — lag vs lead measures
Lag measures tell you what already happened. Lead measures tell you what is happening upstream, where you can still act. When the outcome takes 15 years to move, you need leading indicators that are visible in year two.
Step 5 — Test honestly
We’re using the wrong types of measures
Process measures (compliance rates, audit scores, training completions) answer “are we doing what we said?” Outcome measures answer “has the result changed?” Most programmes answer Question 2 of the Model for Improvement with a process measure.
Step 5 — Test honestly
We’re not allowing enough time to overcome lags
A structural intervention takes time to produce a detectable change point. Declaring failure before the lag has elapsed, then replacing the intervention, resets the clock — and makes it impossible for any signal ever to appear. Deming called this tampering.
Step 5 — Test honestly
We haven’t got a multi-step plan with measures appropriate to each stage
Theory of Change (Carol Weiss, 1995): complex programmes fail because the intermediate steps are not made explicit. Each mini-step in the causal chain needs its own measure and its own Bootstrap CUSUM test. A sequence of change points is the proof the theory is working.
Step 1 — See the pattern
We’re not understanding variation — we can’t tell signal from noise
Common cause variation is the normal output of a stable system — do not react to it. Special cause variation is a genuine signal — investigate it. Bootstrap CUSUM distinguishes the two with statistical rigour. Reacting to common cause as if it were special cause is tampering.
3
Not executing the intervention well enough
Visibility Loop Step 4
Step 4 — Design the intervention
We haven’t executed the intervention well enough — it didn’t reach the structural level
A mandate is not a change point. ENFit was mandated. Rollout is incomplete. A Level 3 intervention on paper only becomes a Level 3 intervention in practice when Bootstrap CUSUM shows a change point in the outcome series.
Step 4 — Design the intervention
We’re not appreciating contradictions — two things we need seem to conflict
The conflict is not a problem to be managed. It is a hidden assumption to be dissolved. The Evaporating Cloud names the five-box structure and surfaces the assumption. TRIZ separation principles design the structural fix that makes the conflict disappear.
Step 3 — Name the belief
We’re not realising the cause might be a wrong mindset
Level 3 Deep: the belief kept by the people with authority to change the structure has never been named explicitly. “Safety is a behavioural problem.” “Delays are inevitable because customers are complex.” No technical fix dissolves a belief. Only naming it and testing it against evidence can.
Step 5 — Test honestly
We’re not anticipating unintended consequences — the balance measures are moving the wrong way
Every intervention that improves one measure risks degrading another. Balancing measures must be designed alongside outcome and process measures before the intervention starts — not noticed after the fact when the damage is done.
4
No idea how to improve — where to start
Visibility Loop Steps 1 & 2
Step 1 — See the pattern
We have no idea how to improve — where do we even start?
Start with who is already succeeding at what you are trying to do. Bright Spots make visible the structural conditions of exceptional performance in systems where most performance is poor. What are they doing differently — and why does it work?
Step 2 — Find the structure
We’re not appreciating inertia — why change is so hard to make stick
The thirteen visibility traps explain why organisations resist structural change. Trap 13 (the success trap) is the deepest: the practices that produced past success are embedded as the way things are done, and are now producing the wrong outcomes. The assumption has never been named.
Step 1 — See the pattern
We don’t know what is generating so much demand — we’re overwhelmed
Failure demand: demand generated by the system’s own failures. In most organisations, 30–60% of demand is failure demand — invisible until you measure it separately. Reducing failure demand reduces total demand and frees capacity for genuine value work.
Step 1 & 2
Too complicated — we don’t know which problem to solve first
The constraint determines which problem to solve first. Everything else is subordinate to it. Goldratt’s five focusing steps: identify the constraint, exploit it, subordinate everything else, elevate it, find the next constraint.
5
Statistical and measurement traps
Visibility Loop Steps 1 & 5
Step 1 — See the pattern
The results improved after our intervention — but did we cause it?
Regression to the mean: extreme results naturally move back toward the average over time, regardless of any intervention. Without a pre-committed Bootstrap CUSUM prediction, natural regression looks like improvement and gets attributed to the programme.
Step 1 — See the pattern
Things seemed to improve when we started measuring — but they haven’t stayed improved
The Hawthorne Effect: performance changes when people know they are being observed. The change is real but temporary — it is not a structural change point. Bootstrap CUSUM distinguishes a Hawthorne response from genuine structural improvement.
Step 5 — Test honestly
Our measures keep getting gamed — people hit the target and miss the point
Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure. Bootstrap CUSUM on the underlying outcome — not the target metric — is the honest test. Outcome measures are much harder to game than process measures.
Step 1 — See the pattern
The same chart tells different stories depending on who presents it
Run charts, SPC X-mR charts, and Bootstrap CUSUM answer different questions from the same data. The choice of chart determines what is visible and what is hidden. Three different charts from 488 weeks of the same clinical data tell three very different stories.
6
NHS-specific: PSIRF and patient safety
All six Visibility Loop steps
Steps 1–6 — All steps
We have PSIRF implemented but don’t know how to demonstrate systemic learning
PSIRF requires demonstration of system-level pattern learning — not individual event investigation. Bootstrap CUSUM on your incident rate series is the analytical test. A downward change point is honest evidence. A flat line is the honest finding that the structural cause has not been reached.
Step 2 — Find the structure
We keep investigating the same type of incident and it keeps happening
17.5 wrong-route Never Events per year. 15 years. No change point. The investigations were not wrong. The structural cause — physical connector interoperability — was never addressed because it sat outside the investigation room’s authority.
Steps 1–5
A&E performance has not improved despite years of policy interventions
184 months of NHS A&E data. Four stages of structural decline. Zero structural improvement change points. 25 years of Level 1 and Level 2 fixes applied to a Level 3 problem: delayed transfer of care consuming 13,700 beds per day, sitting outside the A&E boundary in social care budgets the NHS does not control.
Step 1 — See the pattern
Some departments are doing much better than others — why?
Bright Spots analysis identifies what the high performers are doing structurally that the others are not. Watford General’s corridor care model dissolved four visibility traps simultaneously. The structural conditions are transferable — even if the whole model is not.
Test your own data
Upload a CSV of your outcome measure. Bootstrap CUSUM will tell you whether a structural change has occurred — or whether what you are seeing is normal variation.
▶ Open the StepChange AnalyzerNot sure where to start? The six-step Visibility Loop →