How to design a measurable marketing experiment
Plan a marketing experiment with one hypothesis, a clear comparison, and rules for measuring outcomes and deciding when to stop.
Free app. Runs on your Mac with the Claude or Codex plan you already have.
What a marketing experiment is
A marketing experiment is a structured comparison that helps you decide whether a change is worth adopting. Start with one decision, such as whether to replace a demo page headline with wording that makes the audience for the product clearer. An A/B test compares two versions. Use the current headline as the control and the new headline as the variant. With random assignment, chance determines which version each eligible visitor gets. Running both groups together helps account for shifts in traffic or demand.
Good marketing test design includes a preregistration, a written plan you save before anyone sees outcomes. It defines who counts, how to measure success, when to stop, and how to handle disruptions or inconclusive results.
What it does for the business
- Choose which change earns rollout timeUse the estimated gain to decide whether a new message merits a wider rollout. If the gain is too small to justify the rollout, direct design and engineering time to a different change.
- Protect the sales team's timeA demo page can bring in more bookings and leave sales with more leads to qualify. Set an acceptable range for booking quality so you can judge the extra appointments alongside the work they create.
- Set a realistic date for the decisionEstimating traffic shows whether you can gather enough evidence before the rollout decision is due. You can then rescope the test, move the date, or use a pilot to learn how people respond.
- Give the next test a starting pointA saved plan and result show who saw the change and what happened. Use that record to refine the next hypothesis and identify where a different audience needs fresh evidence.
Marketing experiment, step by step
Name the decision and scope the experiment
A useful experiment begins with a decision you expect to make.
What to do
- Name the decision and its owner. For a demo page, decide whether to replace the headline for eligible visitors from a campaign.
- State the hypothesis as a prediction. A headline that names the intended buyer may increase completed bookings because suitable visitors can recognize that the offer is for them.
- Describe who is eligible and how they arrive. Choose campaign sources and dates that match where you would use the result, and check for planned promotions or site changes that could affect the comparison.
- Gather the current page, campaign brief, booking event definition, and any recent traffic and booking data for that audience. Record each source's date and coverage.
Why
A headline can work differently for people who already know your brand and people discovering it through an ad. Defining the rollout audience helps you choose relevant sources and see where you need more evidence.
Define the comparison and what counts
Someone reviewing the plan should be able to tell exactly who enters the test and what event counts for them.
What to do
- Write the exact headline for each group. Keep the form, offer, and booking flow the same.
- Choose the unit of randomization, meaning what gets assigned once. Use a stable visitor identifier if available. If coworkers are likely to share what they see, consider assigning whole accounts and accounting for that in the analysis.
- Define the primary metric in terms that can be counted. For booking rate, divide the number of eligible assigned visitors who complete a booking during a fixed period after assignment by all eligible assigned visitors in that group. Count each visitor once.
- Decide eligibility and exclusions before looking at outcomes. Set rules for internal traffic, visits without a stable identifier, and duplicate events. Check how either group could lose data under those rules.
- Test the instrumentation, which is the tracking that records assignment and an outcome. Follow the booking flow in both variants. Confirm that a booking links to the assigned visitor and that reloading the confirmation page produces a single counted booking.
Why
Repeat visits can expose the same person to both headlines if assignment resets each session. Counting only people who reach the form can also distort the comparison if the headline itself affects who goes that far. A stable assignment and a denominator defined before outcomes keep the comparison tied to the question you set out to answer.
Write the decision and stopping rules before launch
The test plan should tell you when to stop before a result gives you a reason to stop.
What to do
- Use the baseline outcome rate for comparable traffic and define the smallest improvement worth acting on. Estimate sample size for that improvement using the planned assignment, analysis, and thresholds for false positives and the chance of detecting that improvement. Check whether enough eligible traffic can arrive before the decision deadline.
- Set a stopping rule with an enrollment target, a minimum run period that reflects normal traffic patterns, and a latest acceptable end date. Set when to follow up on the last enrolled visitor so both arms have the full booking window. Define how you will report a test that hits its deadline short of the target.
- Pick guardrail metrics that capture reasons to pause or decline rollout, such as a booking error rate or a drop in the share of booked demos that qualify. Specify the data source, acceptable limits, and the action each limit triggers. If qualification takes time, plan a follow-up window for it.
- Choose the analysis in advance. Compare the primary outcome between groups and report the estimated difference with an uncertainty interval from the planned analysis. Set an interpretation rule that considers how large the gain is, how uncertain it is, and whether the guardrails passed.
- Document foreseeable disruptions, like visitors seeing both variants or an analytics outage. If the team wants to test several headlines or draw conclusions for several audience segments, account for the extra comparisons in advance or treat those findings as exploratory.
Why
Checking results repeatedly and ending as soon as a conventional threshold is crossed increases the chance of a false positive under a fixed analysis plan. Missing baseline data also makes any claim about how much traffic you need hard to justify. You can still agree on the hypothesis, outcome, and tracking while you gather enough evidence to size the experiment.
Review the plan against the evidence
Review the plan as someone who will have to apply the same rules after the test ends.
What to do
- Ask a reviewer to reconstruct the setup from the written plan. Have them trace one eligible visitor from assignment to a qualifying booking and confirm how the same visitor would be counted if they returned later.
- Reconcile the inputs used to estimate traffic and baseline performance with the original sources. Check the date range, the eligible population, the numerator and denominator, and whether recent changes make that baseline a poor fit.
- Test the written rules against awkward cases. Decide what happens if tracking fails in one group, a promotion changes the audience, or the end date arrives before enough people enroll. Investigate any unexpected difference between planned and observed group sizes before interpreting outcomes.
- Resolve material gaps before launch. If a missing baseline prevents sizing, assign someone to collect it.
Why
A review exposes gaps while the setup is still easy to change. If the booking system records a booking but the team cannot connect it to the original assignment, the primary measure needs a tracking fix before traffic runs through the test.
Hand off the plan and record the decision
A preregistration is useful when the person running the test can follow it and the person reading the result can audit it.
What to do
- Save a dated version of the reviewed plan and give the setup, tracking, and review work to named owners. Confirm who decides whether to start and who can pause the test for a tracking failure or guardrail breach.
- Record changes and their timing if something forces a deviation. Keep the original alongside the revised plan so a reader can see what the team knew before making each change.
- After the measurement window closes, record eligible counts, the outcome difference and its uncertainty, guardrail results, and any deviations. Use the decision rule to recommend rollout, retaining the control, or gathering more evidence.
Why
Keeping the plan with the result lets a reader reconstruct the comparison and judge whether it applies to a future campaign.
Common mistakes
- Changing the offer and the headline togetherA bundled change measures the effect of the full package.
- Choosing success from the dashboardPicking the best-looking metric after the test turns a chance finding into a rollout recommendation.
- Treating sessions as separate visitorsReturning visitors can enter both arms or count several times when the unit of assignment is a session.
- Closing the test on a promising dayEnding a fixed-plan test when the trend looks favorable undermines the error rate assumed by its analysis.
- Calling a wide range of estimates proof of no effectAn inconclusive result can still be consistent with gains or losses large enough to matter.
What a good result looks like
- A named decision owner and a hypothesis tied to one change and one audience.
- Exact control and variant definitions, assignment unit, eligibility, and exclusions.
- One observable primary outcome with its numerator, denominator, and measurement window.
- Traffic and baseline evidence or a documented sizing gap, plus guardrails and explicit stopping conditions.
- A dated analysis and decision rule with a version history and a record of unresolved issues.
And then what? Using the result
- Put the assignment and event definitions into the launch brief so engineering and analytics can check the same setup.
- Run the end-to-end tracking check in both groups before enrolling visitors.
- Use the planned decision rule after follow-up closes and record the evidence alongside the decision.
- Carry open questions into the next campaign brief, with the audience and conditions that produced them.
When to run it
- Before replacing a landing page messageTest the proposed text with the audience that will see it if you roll it out.
- Before expanding an email offerDecide how you will compare outcomes and handle overlap if some recipients share an account or can forward the offer.
- When booking quality is as important as volumeSpecify the sign of a poor-quality booking and give sales enough time to mark qualification before deciding.
Questions people ask
How do you design a marketing experiment?
Write a hypothesis tied to a decision, specify a control and a variant, and record how you will assign participants, measure the outcome, and stop. Use current traffic and baseline evidence to check whether the plan can answer the question in time.
What should an A/B test plan include?
An A/B test plan should include the hypothesis, the two versions, eligibility and assignment rules, tracking, one primary metric, guardrails, stopping conditions, and the analysis method. Add who owns each part so the plan can be carried out.
How do you choose a primary metric for a marketing experiment?
Choose the most decision-relevant outcome you can observe reliably for all eligible assigned units during a fixed window. Define exactly which events go into the numerator and which units go into the denominator.
How do you estimate sample size for an A/B test?
Use the baseline rate, the smallest effect worth detecting, the planned allocation, and the error and detection thresholds for your chosen analysis. Base feasibility on eligible traffic and use a calculation that accounts for clustering if you assign whole accounts.
When should you stop a marketing experiment?
Stop enrolling when the prespecified rule is met, then wait until participants have had the time to convert that the measurement plan allows. Document earlier stops for tracking failures or guardrail breaches and apply the reporting rule you set for them.
What can you do if you have too little traffic for an A/B test?
You can extend the planned run before launch if the campaign and decision deadline allow it, choose an audience with enough relevant traffic, or run a pilot to learn about tracking and customer response. Interpret a pilot within the questions it can answer.
More jobs in Marketing
Design a measurable marketing experiment, on your own Mac.Free app. Runs on your Mac with the Claude or Codex plan you already have. Your files stay on your computer.
Download for Mac