THE TEST AND LEARN PLANNER
Should you run that test?
A test you can't rely on is worse than no test at all — it burns budget and settles nothing. Most testing advice tells you how to run a test; this tells you first whether you should, and if so, how. Answer a few questions and get a straight recommendation — including how, in many cases, you can find your answer without spending a penny.
The thinking behind it
The Test Planner is built on a framework we use with clients to keep testing watertight — intentional, statistically sound, and always laddering to action. Here's the thinking in full.
1. Define what you actually need to learn
Before you spend, get precise about what a result would change. A paid test isn't always the answer — sometimes the insight already exists, or can be found for free.
Run an insight-gap analysis.
Map what's already known before defining what you need to learn. A workshop works well: get people to name what they think they need to learn to drive impact.
If the answer's obvious, find the real blocker.
Is it missing data, or stakeholder opinion? If it's opinion, data may be what cuts through. If it's alignment, a conversation is cheaper than a test.
Ask around first.
The answer often already exists in the organisation. Checking with colleagues takes questions off the table fast.
Prioritise gaps by ROI.
Score them on impact (high/low) and time-to-value (short/medium/long), and factor in the lift to act on them. High-impact findings you can apply immediately come first.
Check it can't be gathered for free — before paying.
See the method section for the free routes. For audience-led questions, this may also mean starting with existing insight, social listening, CRM data or audience intelligence before commissioning a paid test.
2. Choose the right method
Match the method to the kind of insight you need — don't default to paid media.
Paid surveys
Quant at scale, with some qual. Good breadth.
Focus groups
Emotion and the "why", the way only human interaction can reach.
Paid media
Themes, messages, journeys and growth against real behaviour — but expensive if you need conversion significance.
Test opposites, not neighbours.
Don't squint at fractional uplift.
Don't pit a low ask against a medium one and squint at fractional uplift. Go low versus high, see how each end performs, then test a level deeper.
Right-size budget and significance.
Use a statistical significance calculator to find the volume of results you need, then cap spend there — don't triple the budget for the sake of it.
If significance is too pricey, use a proxy.
Higher-funnel metrics like journey starts or click-throughs cost a fraction and give you enough to build from.
3. Give the test space, but be ready to react
Let it run clean — but don't let it run broken.
QA fiercely before launch.
A test that isn't tracking properly gives you nothing you can act on.
Once live, don't interfere.
For paid media, touching the campaign, creative or ad set sends it back into learning and erodes reliability.
Monitor tracking, not performance.
Check data is feeding through; fix and relaunch early if it isn't. Don't panic at low early numbers — campaigns usually pick up after they exit the learning phase (often ~30 results).
Pause at 25%, upscale on a winner.
A quarter of the way through spend with little to show? It probably won't turn around — conserve the rest. Seeing strong, reliable return? Grab your result data, then scale to drive outcomes.
4. Communicate results so they land
Insight that isn't understood doesn't get actioned.
Identify your audiences.
Map who needs to buy in, and what you need them to think, feel and do. Some will be glad of the steer; others need convincing — that's where your upfront rigour pays off.
Respect data literacy.
Don't alienate people with complex charts. Lift out the clear story, answer the "so what?", and make sure every insight ladders to action.
Showcase it widely.
A culture of testing needs buy-in beyond the core team. Share findings, keep insight out of silos, and frame the long-term payoff so the next test is easier to get over the line.
5. Evolve and iterate
Testing only matters if it changes what you do.
Put findings into practice.
Roll the insight out wider and map the impact — testing for its own sake is waste.
Go deeper before moving on.
There's often more to learn from a strong result. If a follow-up test would beat starting fresh, do that first.
Prove the uplift.
Once enough time has passed, quantify the gain from rolling the finding out — then shout about it. Demonstrated return is what secures buy-in for the next round.
The Test Planner
Should you run that test?
A test you can't rely on is worse than no test at all. Answer a few questions and we'll tell you whether to test, how — or how to get the answer without spending a penny.