Interpreting your StepChange chart

A practical way to look for change

Start with the right measure and a clear question: what change would be useful? Then use these four steps together with what you know about the process.

1. Look at the run chart

Plot the observations and note when the process change was introduced. Six consecutive points above or below a suitable baseline median suggest a possible shift worth investigating. Look at when the sequence began and whether its direction and size are useful. It is a signal, not proof that chance played no part.

Look at the spread as well as the average. Is the measure fairly consistent, or does it move around a lot? For a particular improvement, wider ordinary variation can make a change harder to detect, so more observations may be needed before a sustained shift becomes clear. Consider the improvement sought, the available time and the pattern in the data; no detected shift does not prove there was no benefit. Check gaps, unusual values, trends, seasonal patterns and changes in recording. Consider whether the overall measure combines different teams, shifts, patient groups or pathways. Compare groups only where definitions, case mix and numbers make that comparison useful; differences are clues, not proof of cause. More on inspecting variation →

Keep three dates separate: when the process changed, when performance may have started to change, and when enough observations had accumulated to give a signal. They can differ.

In this analyser: the run-chart median is calculated from all observations in the selected period, rather than a separately chosen baseline. Check whether that is a suitable reference; record the intervention date separately. Points on the median are ignored when counting a run. Gaps can mean six consecutive recorded points are not six consecutive periods.

2. Look at the green CUSUM line

A major bend or turning point can draw attention to a possible change. Compare its timing with the run chart and what you know about the process. Both views use the same data, so agreement is useful but not independent confirmation. Not every shift produces a clear peak or trough in the overall CUSUM.

3. Explore the step-change chart

Use it to explore possible stages, their average levels and when changes might have begun. A 95% or 99.7% setting changes how demanding the test is; it does not prove that a particular change could not be due to chance. Results should be interpreted with care and do not prove what caused a change.

4. Check whether the findings are dependable

Look for missing data, changes in recording, unusual values, skewness, trends, seasonality and relationships between successive observations. A skewed distribution is not automatically a problem, but it and the other features may need closer review. The optional repeatability and autocorrelation checks can provide clues; they do not verify all assumptions.

Then decide and keep reviewing. Is the observed change worthwhile? What else changed? Consider risk and any balancing measure, decide what to do next, and continue monitoring.

Understand the optional result checks →

Read the Step Change Analysis view

• Red points and line—your observations. Read these against the left-hand scale. Look at the dates, units, gaps and unusual values.
• Blue line—stage averages. Each horizontal section shows the average for one period identified by the analysis. A step joins periods with different averages. Higher is not always better: fewer late deliveries may be good; fewer completed orders may not be.
• Green line—CUSUM. Read this against the separate right-hand scale. It adds up each observation’s difference from the whole-series average: it rises for values above that average and falls for values below it. A rising CUSUM is not automatically improvement. Turning points help the method locate possible changes; not every bend is a detected change.

If you select Show 3SD Limits, the dashed bands show each stage average plus or minus three stage standard deviations. They describe spread; they are not targets, confidence intervals for the average or the same limits as the separate X-mR chart. Stage boundaries are estimates, and short stages can give unstable SDs. A small SD alone does not establish stability.

Wayne Taylor’s conclusions on change-point analysis

A complementary tool. Taylor explains that change-point analysis does not replace control charts. Control charts are valuable for monitoring a process as new measurements arrive and for drawing attention to unusual individual points. A change-point analysis looks back over accumulated data to help explore when the average level may have changed. The charts can be used together.

For a one-time review of historical data, especially a large series, Taylor argues that change-point analysis can be particularly useful. In his conclusion he describes the following potential advantages of his approach:

1. Finding smaller sustained changes. It may reveal a continuing change in level that is difficult to see on an individuals control chart.
2. Describing changes. It can identify more than one possible change and estimate when changes took place. Taylor’s method also reports confidence levels and intervals for change dates.
3. Addressing false detections. Taylor says his approach controls a “change-wise” error rate, contrasting it with the point-wise error rate of control charts. This is a claim about his method and its assumptions—not a guarantee for every implementation or dataset.
4. Handling unusual values. Taylor describes his approach as robust to outliers and notes that using ranks can provide additional robustness. An unusual value should still be checked; no method makes data errors irrelevant.
5. Flexibility. Taylor discusses applying the approach to different types of time-ordered data, including data that do not follow a normal distribution. That does not mean every dataset is suitable: the assumptions and the way the measure was collected still matter.
6. Making a long series easier to review. The analysis can summarise a series as stages, helping a user explore several possible changes and their timing rather than interpreting many individual signals one by one.

Limitations Taylor highlights

• Isolated unusual points: change-point analysis focuses on changes in the level; a control chart can complement it when individual unusual observations matter.
• Different results on repeat runs: random bootstrap samples can produce slightly different answers. More shuffles can reduce calculation-to-calculation variation, at the cost of time; they do not fix unsuitable data or assumptions. Taylor’s paper recommends at least 1,000 bootstraps for his examples and discusses larger numbers where practical.

Taylor also suggests software could check assumptions such as independent errors and alert users to potential problems. Such checks can be helpful, but a simple diagnostic cannot prove that observations are independent.

What this means for this analyser: Taylor’s conclusions describe his approach, not a guarantee that this analyser has identical properties. The analyser does not provide confidence intervals for change dates. Its error rates have not been fully validated for all data, and its repeatability and autocorrelation checks are exploratory. A detected step does not establish the cause of a change or guarantee that it will last.

Reference: Wayne A. Taylor, Change-Point Analysis: A Powerful New Tool for Detecting Changes, Variation.com, conclusion, pp. 17–18.

Use the Stage Summary

Check the stage dates, Mean (average) and SD (spread around that average). Compare the size of the difference with what matters in practice—not just whether a step appears.

Change % compares a stage average with the previous stage. It is a relative change, not a percentage-point difference.

Individual stage confidence scores are not shown: a score inherited from an earlier split is not a separate test of the final stage. The selected confidence setting is not the probability that an intervention worked.

What does the result mean?

• More than one stage: the analysis identified possible average-level changes at the selected settings. Check direction, size and timing. Investigate what changed around those dates, including data definitions and outside influences. Continue monitoring; a detected step does not prove permanence or cause.
• One stage: no split was detected with these data and settings. This does not prove stability, no benefit or no problem. The effect might be small, gradual, too recent or hard to detect—or the intervention may not have helped.

Before deciding what to do

Check missing records, date order, definitions and denominators. Seasonal cycles, trends and links between successive observations can make the shuffled-order comparison misleading; the method must suit the data.

Record the period and settings used. Do not keep changing them until a preferred result appears. Errors or inadequate data need attention before interpreting the output.

Compare timing with the intervention and its expected delay, check implementation and possible harm elsewhere. Do not declare success—or automatically wait or launch another initiative—from the chart alone. Urgent safety concerns still require action.

Use the optional checks

If you need more confidence in a result, try the repeatability check and the check for related observations. Differences between runs or possible dependence are reasons to investigate, not automatic proof that the whole result is wrong.

Understand the checks and decide what to do →

Choose the next step

• Possible change: check its size, direction, timing and practical importance. Investigate competing explanations and whether the result continues; do not attribute it to the intervention from timing alone.
• No split detected: check data quality, the measure, implementation, expected delay and whether the method could detect the effect you expected. An unacceptable result still needs attention; consider whether the explanation or intervention needs revision.
• Data or method problem: correct errors or choose an appropriate analysis before drawing conclusions.

For the variation principles, see Understanding variation. To reconsider the explanation or intervention, return to Right problem? Right intervention? Right measures?

What’s your next useful question?

What explanation could account for this result—and what evidence would help us check it?