Experiments
An experiment shows each person one variant, the same one every time, and measures which reaches your goal more.
Who takes part. Each experiment has a *unit*, fixed when you make it:
- device — visitors who gave consent (
consent(true)); assignment is stored on their device. - user — people signed in through
identify; the same variant on every device.
Everyone else sees the original and is not counted. Cookieless visitors are never in an experiment: their daily id would change the variant they see from one day to the next.
Variants.
- *Code*: branch in your own code —
if (DeltaAnalytics.variant('checkout-cta') === 'big') …. - *Redirect*: send the variant to another address (query string kept).
- *Visual*: change text, styles, attributes, or hide elements with the visual editor (Pro), which opens your page with an editing panel.
Preview a variant. On the experiment's page, *Preview* makes a link for each variant: a page of your site opened with ?dba_qa=<experiment>:<variant>. The browser that opens it sees that variant, whatever its consent or the split, before or while the experiment runs; nothing it does is counted, and a bar at the foot of the page says so. The preview lasts for that tab, across its pages, until you press *End preview* on the bar. A flag is previewed the same way, on or off: ?dba_qa=new-checkout:1.
Feature flags (Pro and higher) turn on for a share of people (the rollout), and can be targeted: *only signed-in people* keeps everyone your site hasn't identified out, even at 100%; and *always on for these people* names up to 500 people by your own ids for them, in whatever the rollout. The page is only ever sent those ids scrambled. Our server libraries take the flag as /api/v1 lists it and do the same with the people named: flagOn(flag, user.id).
No flicker. For visual and redirect tests, add this before the tracker; the page shows as soon as the variant is applied, or after 300 ms at most:
<style>.dba-hide{opacity:0!important}</style>
<script>document.documentElement.classList.add('dba-hide');setTimeout(function(){document.documentElement.classList.remove('dba-hide')},300)</script>Server-side assignment. Our packages assign exactly as the tracker does (assign(experiment, unit)), so you can render the right variant on the server with no flicker at all. docs/experiment-vectors.json holds the test cases.
Reading results. Results use a sequential test, so looking every day does not inflate false wins; a result is called when the adjusted p-value is under 5 %. Beside it: the chance each variant beats the original, the expected loss of choosing it, and revenue per person when the goal has revenue. If the split is far from the weights you set (a *sample ratio mismatch*), the verdict is hidden until you find out why — it usually means something besides the experiment decides who sees what.