Shopify Product Page A/B Testing: The Sample Size Formula Before You Trust a Result (2026)

How long you need to run a product page test depends on two numbers most sellers never pull before launching one: their baseline conversion rate and how big a lift they actually care about catching. Skip that math and a test gets judged the way most dropshipping stores judge them — glance at the dashboard after two or three days, see one variant ahead, flip the toggle. At the traffic most stores run, that gap is almost always noise wearing a winner's jersey.

The tell is in the numbers once you look. A store doing 40 sessions a day per variant at a 3% baseline converts roughly one order a day, sometimes zero, sometimes two. A single extra sale on a Tuesday swings the daily rate by 100%, and it has nothing to do with the headline you changed. Call a test on that kind of sample and you are not reading a result, you are reading last Tuesday.

What follows: why most product-page tests get called before they mean anything, a sample-size formula you can run on your own numbers instead of trusting a tool's black box, what that math actually means in days for a small store versus a bigger one, a grid for reading a test honestly while it's still running, and the contamination risks the formula itself can't catch.

Why most dropshipping product-page tests get called too early

The failure mode is always the same: open the dashboard on day two, see variant B converting 20% higher than A, switch everything over to B. At low daily traffic that 20% gap is built almost entirely out of random order timing, not out of anything the page did differently — the same two variants run again for another two days and the lead flips.

Dropshipping stores are more exposed to this than a high-traffic retailer because the sample fills in so slowly. A 3% baseline rate at 40 sessions a day per variant produces roughly one purchase daily; a swing from zero to two orders reads as a 100-200% conversion move on a dashboard that has no idea it is looking at a coin flip.

The fix is boring and that is exactly why most sellers skip it: decide the sample size before launch, not while watching the numbers move, and resist touching the test until that number is reached or a result crosses a threshold big enough to trust early — covered in the grid further down.

The sample-size formula: how many visitors you actually need per variant

This is the same approximation behind most commercial split-testing calculators, just made visible so you can run it on your own baseline instead of trusting whatever a plugin decided counts as "significant." For a two-proportion comparison at roughly 95% confidence and 80% power, the number of visitors needed per variant is:

Run your own two numbers through it. p is your current product page conversion rate as a decimal — 3% is 0.03. d is the minimum lift, in absolute conversion-rate points, that you actually care about detecting — if you only want to know about a move from 3% to 3.75%, d is 0.0075, not 0.75.

• n per variant ≈ 16 × p × (1 − p) / d²
• p = your current baseline conversion rate, as a decimal (3% = 0.03)
• d = the minimum detectable effect you care about, in absolute conversion-rate points (a move from 3% to 3.75% = 0.0075)
• This is an approximation, not an exact calculation — round the result up rather than down, especially at a low baseline rate

See the ads that are winning right now

Turning sample size into a number of days

Run the formula on a 2.5% baseline where you want to catch a 20% relative lift — a move to roughly 3%, so d = 0.005. That comes out to n ≈ 16 × 0.025 × 0.975 ÷ 0.005² ≈ 15,600 sessions per variant, 31,200 total. At 300 total sessions a day split evenly, that is 150 per variant per day — about 104 days to reach the sample. For almost every small dropshipping store, that lift is simply not catchable in a timeframe anyone will wait for.

Widen the lift you are trying to detect and the picture changes fast. Same 2.5% baseline, now looking for a 40% relative move (to about 3.5%, d = 0.01), and n drops to roughly 3,900 per variant — 7,800 total. At the same 150 sessions a day per variant, that is about 26 days, a test most stores can actually finish. The lesson is not that your test was wrong, it's that the question you asked it was too subtle for the traffic you have.

As a rough guide at a 2.5-3% baseline and a lift worth actually catching (roughly 25-30% relative, needing something like 6,000-7,000 sessions per variant): around 1,000 sessions a day per variant clears that in under a week, 300 a day per variant takes roughly three weeks, and under 100 a day per variant pushes past two months — at that volume, test only changes big enough that you'd expect a much larger lift, not a subtle one.

The decision grid: ship, kill, or keep running

Read the test against the sample target you set before launch, not against how convincing today's gap looks.

• Ship: full sample reached, and the gap has held at or above your minimum detectable effect for the last several days of the run, not just the final one
• Keep A, test something bigger next: full sample reached, but the gap sits inside the minimum-detectable-effect band — at this traffic level there is no real difference between the two pages, and that is a legitimate result, not a failed test
• Early kill, acceptable: under roughly a third of the target sample, but one variant is already trailing by more than double your minimum detectable effect — that gap is wide enough to be a real signal rather than noise, and there is no reason to burn the rest of the traffic proving it again
• Keep running, do not call it yet: under the target sample and the gap sits inside the minimum-detectable-effect band — this is exactly the stage where a "clear winner" today reverses by next week, and it is the single most common place sellers call a test too soon

Running the test clean enough to trust the number

Change one variable per test — hero image, price display, or trust badge placement, not all three. A page that swaps the image, the price layout, and the badge position at once and wins tells you something in that bundle worked, not which part, so the next test starts from a guess again.

Hold the traffic mix still while the test runs. Launching a new ad set, broadening targeting, or shifting budget between platforms mid-test can move the conversion rate on its own — a cheaper, lower-intent audience arriving mid-test will drag both variants down together and has nothing to do with the page.

Run both variants concurrently, never sequentially. A page that ran variant A for one week and variant B for the next is not a controlled test — it is two different weeks with two different traffic mixes, weekday distributions, and possibly two different promotions running, and the comparison tells you nothing reliable about either page.

What the formula can't tell you

A new layout often gets a short-lived bump purely from being different to returning visitors — a novelty effect that tends to fade within one to two weeks. If a result looks strong early, it is worth a second look a few weeks after launch rather than treating day-five numbers as final.

Seasonality and promotions contaminate the read in ways the sample-size math has no way to see. A sale banner, a payday week, or a Black Friday spike shifts conversion rate independently of anything on the page, so a test that straddles one of those events needs a longer run or a different window, not a faster read.

Running many sequential tests at the standard 95% confidence level means roughly one in twenty "wins" is pure chance even when every number checks out — which is why a result on the metric you actually set out to move matters more than a surprising swing on a secondary number buried in the dashboard. Trackira's competitor tracking shows which product-page layouts, badges, and pricing displays the ad-backed stores in your niche are actually running right now, which is a faster way to pick what is worth testing next than guessing from a template library.

How long should I run an A/B test on my Shopify product page?

It depends on your baseline conversion rate, your daily traffic, and the minimum lift you want to be able to detect — not a fixed number of days. Run the sample-size formula on your own numbers first; a store with 150 sessions a day per variant typically needs 3-15 weeks depending on how subtle the lift is that it's trying to catch.

How many visitors do I need for a valid product page test?

Use n ≈ 16 × p × (1 − p) / d² per variant, where p is your current conversion rate as a decimal and d is the minimum lift, in absolute points, you care about detecting. A 2.5% baseline store chasing a 40% relative lift needs roughly 3,900 sessions per variant; chasing a 20% relative lift needs about 15,600.

What is minimum detectable effect and why does it matter?

It's the smallest lift you actually care about catching, expressed in absolute conversion-rate points. Set it too small and the required sample becomes unreachable for most stores; set it to the smallest move that would actually change your decision and the test finishes in a timeframe you can wait for.

Can I trust a test that didn't reach full sample size?

Only if the gap is already more than double your minimum detectable effect and the sample is at least a third of the target — that combination is a real signal. Inside the minimum-detectable-effect band, an early result is noise regardless of how convincing it looks on the dashboard.