A/B Test Results Changelog Block for Vue

Edit TESTS at the top of the file. A test has a name, a metric, the control and variant conversion rates in percent, the visitors in each arm, and the decision (SHIPPED, KEPT or WAITING, with a version when it shipped). The lift, interval, confidence, axis and headline are computed from those.

With the usual normal approximation for the difference of two proportions: the standard error comes from both rates and the sample size, the interval is the lift plus or minus 1.96 standard errors, and the confidence is two-sided. It suits large samples of yes or no outcomes. For other metrics, replace resultOf with your own numbers.

No. A decision is a human call, so it's typed in the data. The bar and the confidence are there to show whether the call was sound: a whisker that crosses the zero line says the test has not separated the arms.

The question each row answers is how far the variant moved from control, so every bar starts at the same zero line and the axis runs the same distance either side, to the widest interval in the table. Gains and losses differ by shape (solid or hatched) and by the arrow and sign in the text, never by colour.

Yes: a native table with a caption, scope on every header and the test as the row header. The bar is decorative, so every number is also text, and screen readers hear whether the lift was better than, worse than or no clear difference from control.

Below md the bar moves under the test's name and the decision under its lift, so nothing scrolls sideways. The axis is drawn left to right in every direction because a lift is a distance from zero; text mirrors.