Short answer: Shopify now has two native testing tools. SimGym sends AI shoppers through an unpublished theme to flag friction before launch, and its results are directional. Rollouts schedules changes and runs A/B experiments on real traffic from your admin; experiments need the Grow plan or higher. Third-party apps still lead for economics: Intelligems for price, shipping and offer tests, profit per visitor and headless stores, and Shoplift for theme, template and price tests on Liquid. Most Plus brands should simulate in SimGym, confirm in Rollouts, and use an app for economics tests.
For years, A/B testing on Shopify meant an app, a script and some patience. In 2026 Shopify added two native options, and in the last month it extended Rollouts to discounts. That changes how a CRO program for a health, beauty or fashion brand should be set up.
Key takeaways
-
SimGym opened to all eligible merchants on March 11, 2026, in AI Research Preview. It works on Liquid themes only and decides comparisons on add-to-cart rate.
-
Rollouts lives at Markets > Rollouts. Launches and scheduling run on Basic and up. Experiments need Grow or higher. Market targeting needs Advanced or Plus.
-
Since June 5, 2026, Rollouts can test themes and checkout and customer account configurations. Since October 2, 2026, it can test discounts on part of your online store traffic.
-
Rollouts experiments report funnel rates (conversion, add-to-cart, reached checkout, bounce). Shopify's docs don't list revenue per visitor or a significance method.
-
Rollouts doesn't support headless or custom storefront checkouts. Intelligems supports headless; Shoplift says it doesn't.
-
Half of AI-referred sessions land directly on a product page, per Shopify. Test PDPs first.
SimGym: AI shoppers before launch
SimGym is a Shopify app that runs AI shoppers through your store. It first appeared with the Winter '26 Edition in December 2025, and on March 11, 2026 Shopify opened the AI Research Preview to all eligible merchants with no waitlist (Shopify Changelog).
How it works
-
Install the SimGym app.
-
Pick an unpublished theme from your theme library and run a simulation against your current theme.
-
SimGym's AI shoppers, which Shopify describes as having human-like profiles, navigate, discover products and add to cart.
-
You get a report on navigation and product discovery, add-to-cart flow and friction points, plus a recommendation on which theme performs better.
Single-theme deep analysis arrived on March 26, 2026. Choose "Analyze a theme," pick a live or draft theme and optionally focus on the homepage, product pages, collection pages or cart. You get strengths, weaknesses and next steps (Shopify Changelog).
Eligibility (Shopify Help Center):
-
Liquid storefronts only. Hydrogen and headless storefronts aren't supported.
-
Shopify Network Intelligence must be activated.
-
The store can't be in private mode and needs at least one product listing.
-
Theme comparison needs your published theme plus at least one draft theme.
Pricing. A monthly free allowance renews on the 1st and doesn't roll over. After that, Shopify charges $0.75 per single-theme simulation and $1.50 per comparison, only when a paid simulation completes.
What SimGym is good for: catching obvious friction before real customers see it. Think a hidden add-to-cart button on mobile, a confusing variant picker or a navigation path that dead-ends.
What it is not: a measure of real conversion. Comparisons are decided on add-to-cart rate, and Shopify says results might differ from actual buyer behavior. Treat a SimGym win as a hypothesis to test, not a winner to ship.
Rollouts: native scheduling and A/B experiments
Rollouts arrived in the admin on March 31, 2026 under Markets > Rollouts (Shopify Changelog). It has grown fast:
| Date (2026) | What Rollouts gained |
|---|---|
| Mar 31 | Schedule theme changes, auto-revert on an end date, A/B test against the current theme with real traffic |
| Jun 5 | Schedule, gradually publish and A/B test themes and checkout and customer account configurations; localized content per market; mutually exclusive experiments |
| Sep 22 | Redesigned setup: pick timing and traffic share and Rollouts sets itself up as a launch, temporary event or experiment; one rollout across theme, checkout and accounts; audience split view; earlier conflict warnings |
| Oct 2 | Discounts: test a new offer against existing offers on a portion of online store traffic, and launch it alongside theme and checkout changes |
Launches vs experiments. Shopify's help center now separates two types. A Launch publishes a change to everyone, all at once or gradually, and you keep or revert it. An Experiment splits visitors between treatment and control. Mutually exclusive experiments let you run several live tests at once, designed so they don't overlap.
Plan requirements
| Capability | Plan needed |
|---|---|
| Rollouts launches and scheduling | Basic or higher |
| Experiments (A/B tests) | Grow or higher |
| Market-targeted rollouts | Advanced or Plus |
| Partial-traffic discount tests | Online Store channel only |
Shopify's help center says "Experiments require the Grow plan or higher" and that Rollouts supports only online store checkouts, not headless or custom storefront checkouts (Shopify Help Center). Grow is Shopify's mid-tier plan, between Basic and Advanced. Shopify hasn't said whether discount experiments carry the same Grow requirement, so assume they do.
What Rollouts measures
Per Shopify's Rollouts analytics page:
| Rollout type | Metrics reported |
|---|---|
| Theme experiment | Conversion rate over time, bounce rate, reached-checkout rate, add-to-cart rate |
| Checkout and accounts experiment | Checkout conversion rate |
| Theme launch | Gross sales, AOV, orders, total sales and sessions over time, conversion rate, new vs returning customers, bounce rate |
Two gaps matter for a serious CRO program. Revenue per visitor isn't listed for experiments, and the analytics page doesn't describe a significance method or confidence level. For a theme or checkout change, funnel rates are usually enough. For a price or offer decision, you want revenue and margin per visitor.
One technical note for app builders: since the October 2026 API, discounts gained a read-only rollouts connection. Discount apps that read only start and end dates may misreport whether a Rollouts-tested discount is live (shopify.dev changelog).
Intelligems and Shoplift: where testing apps still lead
We run both for clients. Here is where each beats native tools, from each vendor's own documentation.
Intelligems (intelligems.io, headless docs)
-
Price tests on products, subscriptions and multi-currency prices, optimized for profit per visitor
-
Shipping tests, including rates and free-shipping thresholds
-
Offer tests: discounts, progress bars, gift-with-purchase, post-purchase offers
-
Content and theme tests
-
Custom goals: clicks, element visibility, scroll depth and custom code
-
Headless support, including Hydrogen, through its headless package
Shoplift (shoplift.ai, FAQ)
-
Theme, template, URL redirect, price and custom JavaScript tests
-
Shipping thresholds through its JavaScript API
-
Subscription testing and personalization by behavior, geography, channel and device
-
Server-side conversion tracking and a native GA4 integration
-
No headless support: its FAQ says custom front ends such as Hydrogen or Next.js aren't supported
Shoplift has argued publicly that Rollouts is useful for deployment management rather than experimentation (Shoplift blog). That's a competitor's view, but it points at the real gap: economics tests and revenue reporting.
Side by side
| SimGym | Rollouts | Intelligems | Shoplift | |
|---|---|---|---|---|
| Traffic | Simulated AI shoppers | Real visitors | Real visitors | Real visitors |
| Best for | Pre-launch theme checks | Theme, checkout, accounts and discount tests; scheduled launches | Price, shipping and offer economics | Theme and template tests on Liquid |
| Headline metric | Add-to-cart rate (directional) | Funnel rates | Profit and revenue per visitor | Conversions, revenue, goals |
| Plan or setup | App; Liquid only | Basic for launches, Grow+ for experiments | App | App |
| Headless | No | No (online store checkout only) | Yes | No |
| Cost | Free allowance, then per simulation | Included in plan | Paid app (see vendor) | Paid app (see vendor) |
Check each vendor's current feature list and pricing before you choose.
A CRO workflow that uses all of them
-
Find the problem. Start with data, not ideas: GA4 funnels, Microsoft Clarity or Hotjar session recordings, support tickets and reviews. For supplements, common issues are subscription vs one-time clarity and dosing confusion. For beauty, shade selection and "will this work for my skin."
-
Build the change on an unpublished theme.
-
Simulate with SimGym. Fix the friction it finds, then rerun.
-
Test on real traffic with a Rollouts experiment. Use market targeting if the change is regional.
-
Escalate to Intelligems or Shoplift when the question is price, shipping threshold, offer economics or revenue per visitor, or when you run headless.
-
Launch the winner with a scheduled Rollout, and log the result with the hypothesis, metric, duration and decision.
Where AI shoppers change the testing plan
More shoppers now arrive from AI assistants already knowing what they want. Shopify reported that in Q1 2026, AI-driven traffic to Shopify stores grew 8x year over year and orders from AI-powered searches grew nearly 13x (Shopify Blog). On its Q2 2026 earnings call, Shopify said half of all AI-referred sessions land directly on a product page (TechCrunch).
Three implications for your test roadmap:
-
Prioritize PDP tests over homepage tests. Half of AI-referred shoppers skip your homepage.
-
Test clarity, not persuasion. These visitors arrive with a specific question. Clear size, dosage, shade and shipping information converts them.
-
Remember the checkout you can't test. Google AI Mode, Gemini, Copilot and Meta's Muse can check out without visiting your site, and Rollouts only tests online store checkouts. Your product data matters as much as your page design. See our Agentic Storefronts settings guide and how to make your checkout agent-ready.
Frequently asked questions
What is Shopify SimGym?
SimGym is a Shopify app in AI Research Preview that runs AI shoppers through an unpublished theme and compares it with your current theme, or analyzes a single theme. It reports friction and directional indicators and decides comparisons on add-to-cart rate. It works on Liquid storefronts only.
What is Shopify Rollouts?
Rollouts is a native tool under Markets > Rollouts for scheduling store changes, reverting them automatically and A/B testing them on real traffic. It covers themes, checkout and customer account configurations, localized content and, since October 2, 2026, discounts.
Can I A/B test on Shopify without an app?
Yes, with Rollouts experiments. Shopify says experiments require the Grow plan or higher, and only online store checkouts are supported, not headless or custom storefronts.
Is SimGym a replacement for A/B testing?
No. SimGym uses simulated shoppers and Shopify says results might differ from actual buyer behavior. Use it to find friction before launch, then confirm with a real-traffic test in Rollouts or a testing app.
Can Shopify Rollouts test discounts?
Yes. Since October 2, 2026, you can test a new discount against existing offers on a portion of online store traffic and launch it alongside theme and checkout changes.
What does Rollouts measure?
Theme experiments report conversion rate, bounce rate, reached-checkout rate and add-to-cart rate. Checkout experiments report checkout conversion rate. Shopify's docs don't list revenue per visitor for experiments.
When do I still need Intelligems or Shoplift?
When you need price or shipping-threshold tests, offer economics, revenue or profit per visitor, custom goals, or tests on a headless front end (Intelligems supports headless; Shoplift doesn't).
Run CRO with XgenTech
XgenTech is a certified Shopify Partner agency founded in 2012 and rooted in New York, with 15+ ecommerce experts and 320+ store launches. Our Shopify CRO team builds hypotheses from Clarity, Hotjar and GA4 data, then runs SimGym, Rollouts, Intelligems and Shoplift tests for Plus brands in supplements, beauty and fashion. Ongoing testing fits into a Shopify retainer, and checkout tests are handled by our Shopify Plus experts. Book a free CRO audit call.
Related reading:
