Playable Ad A/B Testing: Variables, Sample Sizes & Decision Framework

Many UA teams 'A/B test' by uploading several creatives and seeing which does best. Without a controlled variable, that is exposure, not a test. A real test changes one thing, runs long enough to mean something and decides on both install volume and install quality. Over months, the gap between the two habits turns into a large CPI and ROAS gap.

Test One Variable at a Time

If you change both the headline and the background image between variants, you cannot attribute the performance difference to either. Teams that keep to one variable per test learn something; teams that do not just collect numbers. The order to test variables, ranked by typical performance impact:

1. CTA button label: 'Install Now' vs 'Play Free' vs 'Start Playing' vs 'Get It Free'. This single change can swing install rate by 15–25%.
2. Headline copy: game name vs genre hook vs social proof vs FOMO. Second-highest variance source.
3. Background art: gameplay screenshot vs character closeup vs illustrated key art vs abstract brand color.
4. Interaction mechanic (for interactive creatives): tap vs hold vs path, or type of mini-game.
5. Button color: accent vs high-contrast vs brand color. Lower variance than copy changes, but measurable.
6. Layout structure: CTA at top vs bottom, icon placement, text weight.

Minimum Sample Sizes by Network

You need enough events to tell a real difference from noise. The minimum thresholds for playable ad tests:

• Impressions: 5,000+ per variant before drawing conclusions. 10,000+ for higher confidence.
• Installs: 50+ per variant for install-rate comparisons. Below 50, the variance is too high to trust.
• Revenue events (IAP, subscription): 20+ per variant for ROAS comparisons (much harder to reach, longer test windows needed).
• Time: run for at least 7 days. Creative performance often fluctuates in the first 2–3 days as the ML system explores audience segments.

Decision Criteria Beyond CPI

CPI alone is a misleading decision criterion if you're optimizing for revenue. A variant with 10% lower CPI but 20% lower Day 7 retention costs more to acquire the same retained user. Evaluate winners by: CPI (cost efficiency), install rate (creative persuasion), Day 1 retention (install quality), and, if your attribution window is long enough, Day 7 retention or ROAS. For UA campaigns where ROAS is the primary KPI, a 5% higher CPI that delivers 15% better D7 retention is usually the correct winner.

A Worked Example: Testing a CTA Label

The numbers below are an illustration of the arithmetic, not a benchmark. Say your current control reads 'Install Now' and you want to test 'Play Free'. You duplicate the project, change the label and nothing else, export both to the same network, and give them separate ad sets with equal budgets.

• After seven days, the control has 6,000 impressions and 60 installs. Its IPM is 60 divided by 6,000, times 1,000: 10.
• The variant has 6,200 impressions and 74 installs. Its IPM is roughly 11.9.
• Both clear the minimums above (more than 5,000 impressions and more than 50 installs each) so the comparison is worth reading.
• The variant is ahead on volume. Before calling it, check Day 1 retention for both cohorts in your MMP. If the variant's users drop off noticeably faster, the label may be promising something the game does not deliver.

If the variant holds up on retention, it becomes the candidate winner and moves into the scaling phase below. If the two are within a few percent of each other, record the result as 'no meaningful difference' and move to the next variable; a label that makes no difference is a finding too, and it tells you the next test should go further down the list.

Scaling Winners Without Losing Signal

Once a variant wins, don't immediately kill the control. Ramp the winner to 70% of budget, keep the control at 30% for two more weeks as a long-term validity check. Some variants win in the first week because of audience novelty: a creative that looks new to the ML system gets test impressions. If the lead holds for 3–4 weeks, it is real and not novelty.

After confirming a winner, that winning variant becomes the new control for the next test. Test one variable, confirm the winner, make it the baseline, test the next. Kept up over a few quarters, that loop leaves you with far more efficient creatives than you started with.

Testing What Only a Playable Has

The variable list above applies to any creative. A playable has levers a video does not, and the bigger differences are often there:

• The first prompt: how soon the hand or arrow appears, and how obvious the first tap is. See the first three seconds.
• Difficulty: whether the player can lose, and how close the round runs. A creative that is too easy can read as fake; one that is too hard ends before it sells. See difficulty and win-rate design.
• Length: how many interactions before the end card. See length and pacing.
• Ending: a win screen versus a 'so close' fail screen leading into the same CTA.

Test these one at a time, exactly like copy. Changing the difficulty and the length together leaves you with the same attribution problem as changing the headline and the background together.

Common A/B Testing Mistakes With Playables

• Treating the network's own creative optimisation as a split test. When several creatives share one ad set, the system usually shifts spend towards an early leader and starves the rest, so the losing variant never gets a fair sample.
• Comparing variants across networks. A variant on one network and the control on another are different audiences and different auctions; the comparison tells you about the networks, not the creative.
• Editing a live variant. Re-uploading a changed file mid-test resets what you are measuring. Stop the test, record it, and start a new one.
• Peeking and stopping early. Checking daily is fine; deciding on day two is not, for the learning-phase reason given above.
• Testing through a seasonal shift. A holiday weekend or a big store feature changes the audience for both variants at once and can flatter whichever started first.
• Ignoring the in-ad funnel. If both variants install at the same rate but one has a far lower interaction rate, you have learned something about the mechanic even without a winner.

Keeping a Test Log

The loop only pays off if the results are written down. Keep one row per test: the date, the network, the variable, the hypothesis, the two file names, the sample sizes, the result on each metric and the decision taken. After a few months that log is the most valuable creative document the team owns, because it stops you re-testing a label you already ruled out and shows which kinds of change tend to matter for your game. Pair it with a creative brief so every new variant starts from what the log already knows.