Shopify finally built native A/B testing right into the admin — no third-party app, no monthly add-on. It's called Rollouts, and it shipped with the Spring '26 Edition. If you've been paying an external tool to split-test your storefront, this is genuinely good news.
One catch is worth knowing before you get excited. Shopify's native A/B testing isn't the checkout-testing tool a lot of merchants are hoping for. It tests your theme, not your checkout. Knowing exactly what it can and can't do is the difference between a clean experiment and a confident wrong decision.
Treat Rollouts like a full conversion-optimization suite and you'll trust numbers it was never built to give you. It shows how two versions performed, but it won't tell you whether the difference is real. The "winner" on the dashboard might just be noise — and you'll ship it with full confidence.
What Shopify's Native A/B Testing Actually Is
Rollouts is a native split-testing and staged-rollout feature built into the Shopify admin, found under Markets > Rollouts. It splits traffic between your live theme and a copy, and you can ramp the test version from 10% of visitors up to 100% as you gain confidence, according to StorePilot's breakdown of the feature.
The split happens on Shopify's servers before the page loads, so the assigned version is there at first paint. No flicker, no flash of the old design that plagues script-based testing tools. There's no extra fee for it, but the split test itself needs a Grow plan or higher — Basic plans get scheduling and rollback only, per the same breakdown.
Can You A/B Test Your Shopify Checkout?
Not with Rollouts, at least not yet. Rollouts tests theme-level changes only. It can't test prices or discounts, and meaningful checkout edits remain effectively a Shopify Plus capability through checkout extensibility, as StorePilot notes.
So if your goal is testing one-page versus multi-step checkout, or a new express-pay button, native A/B testing won't get you there on a standard plan. What it will let you test is everything leading up to checkout: your product pages, cart, homepage, and theme sections. Those pages decide whether shoppers reach checkout at all, so they're worth testing in their own right.
Start With One Change That Actually Moves Money
The best first test is a single change on a high-intent page. Don't redesign your whole product page and test it as one blob — if it wins, you won't know which change did the work.
Cart and checkout-adjacent pages tend to carry the biggest dollar impact, because those visitors have already shown they want to buy, as Eulav points out in its checkout-testing guide. On a standard plan, your cart page is the closest lever Rollouts gives you to that money. A few starting tests worth running:
- Cart page trust cues — adding a delivery estimate or a returns note near the checkout button.
- Product page buy-box order — moving reviews, shipping info, or the add-to-cart button up or down.
- Homepage hero — a single offer-led headline versus a category-led one.
Pick one. Change one thing. Let it run.
Read the Results Like a Statistician — Because Rollouts Won't
This is where native A/B testing needs you to bring your own discipline. Rollouts shows performance, but it doesn't run a significance test, call a winner, or report revenue per visitor, again per StorePilot. The dashboard hands you raw numbers and lets you decide, which is exactly where most merchants fool themselves.
Three rules keep you honest:
- Wait for 95% statistical significance. That's the standard threshold in ecommerce CRO — it means you're 95% confident the difference isn't random noise, as Contentful's best-practices guide explains.
- Run at least two full weeks. Shorter tests miss day-of-week patterns, and most ecommerce tests need two to four weeks depending on traffic, according to Growth-Engines.
- Gather enough conversions. Aim for roughly 1,000 conversions per version before you trust a result — a few hundred clicks isn't a verdict, as the same Growth-Engines analysis notes.
Run a free significance calculator on the raw counts before you declare anything. And if you don't have the traffic to hit those thresholds in a month, test bigger, bolder changes — small tweaks need huge sample sizes to prove out.
Where Native A/B Testing Still Falls Short
Rollouts is a strong first version, not a finished CRO platform. A few gaps to plan around, flagged in StorePilot's review:
- No winner call or revenue-per-visitor — you interpret the data yourself.
- No audience segmentation yet — you can't target new visitors only, returning customers only, or split by device; traffic is randomized across everyone in your chosen markets.
- Theme-level only — no direct Liquid template edits, no checkout, no pricing tests.
None of that makes it useless. It makes it a clean, free way to test storefront changes, as long as you respect the boundaries.
Your first move this week: pick one change on your product or cart page, open Markets > Rollouts, and run it as a 50/50 split for two full weeks. Don't peek on day three and call it. Judge it against a real 95% significance threshold, not the raw dashboard numbers.
Shopify's native A/B testing will almost certainly grow more capable — segmentation and a real significance verdict are the obvious next steps. Until then, it's the cheapest honest test you can run on your storefront, and it beats redesigning on a hunch.