How to plan and read an SEO split test
Plan an SEO split test with comparable pages and a clear decision rule. Verify the release, compare search clicks, and decide whether to roll out or rerun.
Free app. Runs on your Mac with the Claude or Codex plan you already have.
What an SEO split test is
An SEO split test measures what happens to search traffic when you change one group of comparable pages and leave a second group alone. In this kind of SEO A/B test, you assign pages to groups. You might move the city name forward in the titles of some location pages and keep the existing titles on the control group.
Measure each group's search clicks before and after the change. The control gives you a reference for shifts in demand that may affect both groups, such as a holiday. A decision about the edit depends on a comparison chosen before release, evidence that the change reached search, and enough traffic to distinguish a useful difference from ordinary variation.
What it does for the business
- Make a template decision with evidenceOne template release can affect a whole section of the site. A controlled test helps the client decide where to spend engineering time.
- Catch harmful changes while the release is smallA title rewrite can move a useful detail out of view. Keeping the first release small makes it easier to restore the saved copy if you reach the rollback threshold.
- Give the site editor a clear briefA page list and exact replacement text help an editor estimate and schedule the job. Other teams can plan their work around the measurement window.
- Use an inconclusive result to improve the next testA readout can show whether a few pages drove most of the movement or the edit reached search late. That tells the agency what to adjust before booking another release.
SEO split test, step by step
Choose one change for a comparable family of pages
A test is worth planning when the pages share a problem and draw enough clicks to measure a change.
What to do
- Use the sitemap and menus to count pages in each family, a set of pages that share a template. Note the market and language each family serves.
- Read common and unusual pages in each promising family. Look for a shared problem, such as category titles that put the product type after generic wording.
- Write two or three candidate changes. For each, name the page family, the element to edit, and why it might win organic clicks from unpaid search results. Pick the strongest candidate.
- Use at least 20 comparable URLs per group and about a thousand clicks per group over 28 days as working guides. Check whether a gain worth acting on could stand out against ordinary swings.
- If the usable family is too small or quiet, plan one release on a named group. Compare the 28 days before and after alongside similar untouched pages and, when available, the same weeks a year earlier. Treat the comparison as an indication of direction.
Why
Pages on the same template are more likely to respond to the same change. Reading across the family shows where a rule breaks. A title pattern that works with short city names can crowd out important words on pages with longer names.
Assign the page cohorts and set the decision rule
Write down the conditions for your decision before you can see which group wins.
What to do
- Export clicks by page from Google Search Console for the baseline period, the 28 days before launch. Use finalized dates and keep the market and search filters consistent.
- Exclude URLs edited in the last 28 days, redirected, or hidden from search. Set aside any page accounting for more than a fifth of the family's clicks. Record each reason.
- Rank eligible URLs by baseline clicks. Pair neighbors in the ranking and randomly put one from each pair in each group. With no click data, save a provisional random split you can repeat and rebalance.
- Plan a 28-day follow-up once the change appears in search. Use total group clicks as the main measure. Keep click rate, the share of search appearances that produce a click, and average position as supporting measures.
- Record what gain or loss would justify action, how long it must hold, and when to rerun. List events that would spoil the test, such as a control page edit.
Why
Pairing by baseline traffic helps balance the groups before release. A written rule gives the team a consistent way to handle a strong first week followed by ordinary weeks.
Prepare the exact edit or read the observed results
Read the click totals only after confirming that the planned change reached the intended pages.
What to do
- For a plan, give the publisher the template rule and the exact old, new, and restore text for each test URL.
- For a readout, match the release record to both URL lists. Verify the new text on test pages and the original text on control pages. Check when the change appeared in search.
- Use the same date boundaries and filters for both groups. Mark missing rows and incomplete days so a reader can see where the export leaves gaps.
- For the SEO experiment analysis, show each group's raw click totals before and after. With similar baselines, subtract the control group's change in clicks from the test group's change. Use the calculation set in your plan.
- Apply the written decision rule. Record the case for rollout or rollback, or explain what would make a rerun usable. Note any other edit that could account for the finding.
Why
Measuring from the release alone can mix clicks from the old and new versions. A shift in demand can also produce a large change while both groups move together.
Check the assignments, measurements, and claims
A reader should be able to trace the decision back to the release record and click totals.
What to do
- Ask another reviewer to check the assignments, release record, click exports, and decision rule. If needed, make a separate review pass yourself.
- Reopen a sample from each group and check the recorded text. Recount both groups and confirm each URL has a single assignment.
- Recompute the click totals and the difference in changes. Check that the exports use the planned dates and filters.
- Read the conclusion against what happened. If the release record is missing, the control changed, or the data is too thin for the decision rule, name the limit and give the next step it supports.
Why
One busy URL included twice can change the recommendation. A shift in the mix of searches across groups can also make the raw clicks diverge, so keep any claim tied to what the comparison supports.
Hand off a decision the site owner can act on
A decision is ready to hand off when the owner can see which pages it covers and what action comes next.
What to do
- Lead with the decision and one example of the edit. Point to the URL lists so the site owner can see where the recommendation applies.
- For a plan, give the editor the template rule and text to restore. Name who will confirm the release and collect the click exports.
- For a readout, state the group sizes, which condition led to the decision, and what evidence supports it.
- If a rerun is needed, name the specific fix, such as scheduling it after another team finishes editing the template.
Why
A client may need to pass the decision from the agency to a publisher. A handoff with the reasoning and the exact edit helps both people act on the same decision.
Common mistakes
- Mixing page typesUsing product pages as the control for category pages can confuse a template effect with differences in search demand.
- Changing several elements togetherChanging titles, headings, and internal links in the same release makes it hard to learn which edit contributed to the result.
- Starting the clock on release dayWait for evidence that the change appears in search before starting the follow-up window.
- Counting missing rows as zero clicksA missing observation should stay marked as unknown until you verify what the export covers.
- Reading one good week as the resultUse the window and stopping rules you recorded before the release to limit the effect of a short swing.
What a good result looks like
- One chosen edit with a reason grounded in the pages you read.
- Separate, comparable test and control lists with a reason for every exclusion.
- A dated plan with the primary measure, calculation, decision rule, and events that would spoil the test.
- Exact before, after, and restore text, plus release evidence when reading a completed test.
- Raw clicks for both groups in both windows and a decision another reader can reproduce.
And then what? Using the result
- Reserve the release window with the site owner and name the person who verifies the edit.
- Put an export date on the calendar so you can read both groups once their follow-up data is complete.
- If the test supports rollout, schedule the change for the remaining eligible pages on the tested template.
- Use the saved restore text if the result calls for rollback, then check the affected URLs against the original list.
- If you need another run, put the concrete fix and the revised dates in the next test plan.
When to run it
- Before a template changeA page family gives you a way to test a title or heading rule on part of the site before applying it to more URLs.
- After a documented experimentA release record and comparable exports let you match the observed traffic to the pages that actually received the edit.
- When the site has a small page familyA before-and-after comparison gives the owner a way to observe a worthwhile edit while keeping the limits of the evidence visible.
Questions people ask
How do you run an SEO split test?
Choose one edit, divide comparable pages into test and control groups, and agree on how and when you will measure search clicks. Apply the edit to the test pages and check that it reached search. At the end of the planned window, compare both groups' changes and apply the decision rule.
How many pages do you need for an SEO split test?
Start with a working target of at least 20 comparable pages per group and enough baseline clicks to see a change of practical value. Check how much traffic remains after you exclude recently changed or unusually busy pages. A large group with little traffic can still give an inconclusive result.
When should you start measuring an SEO split test?
Start the follow-up window when you have evidence that search engines show the new text. Use a dated sample of search results to support the start for a title test. If the rollout into search is uneven, record the gaps and consider whether the planned window gives the groups a fair comparison.
How do you calculate the result of an SEO split test?
Calculate the before-to-after change in clicks for each group and subtract the control group's change from the test group's change. Use the raw counts to make the arithmetic repeatable. Choose the comparison in advance, especially if the groups begin with different click totals.
What can you do when a site is too small for an SEO split test?
Make the edit on a named set of pages and compare clicks before and after alongside a similar set you leave alone. Where records are comparable, add the same weeks from the prior year to help assess seasonality. Use the result to decide what to observe or test next.
Which data should you export for an SEO split test?
Export page-level clicks from Google Search Console for the baseline and follow-up periods, keeping the filters and export scope the same. In the Search results performance report, choose the dates, open the Pages tab, and download the CSV. Keep the release record and the group lists with the exports.
More jobs in SEO agencies
Prepare and read an SEO split test, on your own Mac.Free app. Runs on your Mac with the Claude or Codex plan you already have. Your files stay on your computer.
Download for Mac