How the StepChange Analyzer has been tested
What we checked, what we found, and what still needs work.
The first round of testing is encouraging for the cases tested. It does not establish that the analyser is reliable for every dataset. In particular, related observations can make a shuffled-data comparison misleading.
Last reviewed: 6 November 2026. This testing was carried out during development and has not had independent statistical review.
Why test the analyser?
The analyser looks back over observations to explore possible changes in average level. We need to check both that the calculations work as intended and how the complete procedure behaves when the true pattern is known. A chart supports judgement: it does not establish cause, prove that an intervention worked or replace action on urgent harm.
What we checked
- Known calculations: for small examples, every possible ordering was checked, including cases with tied results. The candidate calculation matched the exact comparison in the cases tested.
- Basic behaviour: a constant series and a clearly constructed step produced the expected basic results. Checks also covered calculation precision, settings and the SD bands.
- Simulated data: we tested series with no change, known changes, skewness, trends and related successive observations.
- The interface: a CSV was loaded, both result checks ran, changing settings cleared previous check results, and a mobile view was checked locally.
The corrected worker and matching files were subsequently published. These tests cover particular examples, not every input or future software change.
What did the simulations show?
For the examples below, there was no underlying change. Each series contained 60 observations. The analyser used a 99.7% setting, 1,000 shuffles and a turn length of 5. We counted series with at least one detected boundary.
- Independent normal observations: 3 of 1,000 series showed a possible change.
- Independent skewed observations: 0 of 600 series showed a possible change. Zero observed does not mean the true rate is zero.
- Moderately related neighbours: 33 of 600 series showed a possible change.
- Strongly related neighbours: 331 of 600 series showed a possible change.
The related series were deliberately constructed examples, not estimates for your data. They show why randomly shuffling related observations can give misleading confidence figures. Three signals in 1,000 is too few to estimate a rare false-signal rate precisely.
In a separate example with a real one-standard-deviation shift halfway through independent normal data, a boundary was found within three observations of the true change in 195 of 500 series. The setting, data and change were specific to this example; the figure is not a general detection rate. A real change can be missed.
What the extra checks can tell you
Repeatability runs the calculation ten times with the same data and settings and shows changes in stage counts and dates. Agreement shows consistency across those runs, not proof that a boundary is correct.
Related observations: the exploratory check offers a clue about neighbouring values. Its estimate can fluctuate even for independent observations. A positive result is not a diagnosis, and a small estimate does not prove independence.
More shuffles or repeated runs may reduce calculation-to-calculation variation; they do not fix dependence, seasonality or an unsuitable model. Use the checks alongside the observations and knowledge of the process.
What remains to be checked?
- Broader testing with multiple changes, seasonality, changing variation, short stages and missing data.
- How the recursive search behaves when real changes and additional false boundaries occur.
- Comparison with another method on known cases, using documented settings. Agreement would not by itself prove either method correct.
- Independent statistical review, if available. The first round was carried out by the development team.
Turn Length controls when short segments stop being tested; it does not guarantee a minimum number of observations on both sides of each returned boundary. The analyser does not provide confidence intervals for change dates.
How to use a result
Start with a clear measure, the time plot and a useful question. Check the data and the result, consider its practical importance and what else changed, and keep monitoring. Seek additional help for important or unclear decisions where possible.
Interpret your StepChange chart →
Understand variation →
Return to the analyser →
Further reading
Analyser reference: terms, charts and settings
Wayne A. Taylor: Change-Point Analysis: A Powerful New Tool for Detecting Changes. Taylor’s conclusions describe his method, not a guarantee that this implementation has identical properties.