// execute
A/B Test Design & Significance
Use when the user wants to check whether an A/B test result is statistically significant, or to size a test before running it. Also use when the user mentions A/B test, split test, statistical significance, p-value, confidence level, conversion lift, sample size, or "is this result real / can we call the test." Produces lift, p-value, a significance verdict, and required sample size.
npx skills add sarojkjha/aaj-marketing-skills --skill ab-test-significanceHow to install and run this
Install it in your project:
npx skills add sarojkjha/aaj-marketing-skills --skill ab-test-significanceNo install? Add https://aaj-engines-mcp.sarojkjha10.workers.dev/mcp as a custom connector in Claude, ChatGPT or any MCP client.
https://aaj-engines-mcp.sarojkjha10.workers.dev/mcpThen ask: Use the AAJ ab-test-significance skill on my numbers.
Step-by-step for every agent, Windows, Mac and Linux →Claude Code
/plugin marketplace add sarojkjha/aaj-marketing-skills/plugin install aaj-marketing-skills@aajOpenAI Codex
codex plugin marketplace add sarojkjha/aaj-marketing-skillscodex plugin add aaj-marketing-skills@aajClaude or ChatGPT — no install
Add this as a custom connector:
https://aaj-engines-mcp.sarojkjha10.workers.dev/mcp// when to use it
Use when the user wants to check whether an A/B test result is statistically significant, or to size a test before running it. Also use when the user mentions A/B test, split test, statistical significance, p-value, confidence level, conversion lift, sample size, or "is this result real / can we call the test." Produces lift, p-value, a significance verdict, and required sample size.
// what you give
Visitors and conversions for control and variant (to evaluate) OR baseline rate and minimum detectable effect (to size a test)
// what you get
Conversion rates, relative lift, p-value, significance verdict, and required sample size
Engine spec
// decision it supports
Is the difference between control and variant real or noise, and how many visitors do we need before we can call the test?
// inputs
Visitors and conversions for control and variant (to evaluate) OR baseline rate and minimum detectable effect (to size a test)
// output
Conversion rates, relative lift, p-value, significance verdict, and required sample size
// acts or analyzes
Analyzes onlyMakes no network calls, writes no files and changes nothing in your tools.
// where your information goes
| How you run it | Where your inputs go |
|---|---|
| Skill installed in your agent, or the local MCP server | The engine runs on your machine and AAJ receives nothing. Your AI agent still sends what you give it to its model provider (Anthropic, OpenAI and so on), under your own account's terms. |
| Hosted MCP server | Your agent's model provider, as above, plus AAJ's server on Cloudflare, which receives the engine inputs over HTTPS, runs them in memory and keeps no copy. Cloudflare keeps error logs (no inputs) for up to 7 days. |
No engine needs customer names, emails or other personal data; leave them out.
// limitations
- Uses a pooled two-proportion z-test (a normal approximation) on two arms. It assumes each visitor is counted once, assigned at random and independent of the others, and that the sample size was fixed in advance.
- It reads only the totals it is given. There is no correction for peeking or for testing several metrics or variants, so checking repeatedly and stopping at the first significant result inflates false positives.
- One-tailed mode only asks whether the variant beat control, which halves the p-value; use two-tailed unless the direction was fixed before the test. Plan mode always sizes for two-tailed.
- Plan mode takes the minimum detectable effect in percentage points (5% to 6% is 1), not as a relative lift. An unrecognised field stops the run with an error.
// review before acting
// setup
npx skills add sarojkjha/aaj-marketing-skills --skill ab-test-significancehttps://aaj-engines-mcp.sarojkjha10.workers.dev/mcpAgents listed: Claude Code, Cursor, OpenAI Codex, Windsurf, Cline. Tested and recorded: Claude Code, OpenAI Codex (Windows, ab-test-significance only).
A/B Test Design & Significance
Tell the user whether a test result is real or noise, and how big a test needs to be before it can answer the question. Most teams call tests too early on too little data; this keeps the decision honest with a proper two-proportion z-test.
When to use
The user has an A/B test result to evaluate, or wants to size a test before launching it.
Workflow
To evaluate a result: gather visitors and conversions for control and variant, pick a confidence level (usually 95%), and run the engine. To plan a test: take the baseline conversion rate and the minimum lift worth detecting (the MDE), and compute the required sample size.
Paths assume you installed with
npx skills add. From a clone of this repo, useskills/ab-test-significance/resources/…instead.
node .agents/skills/ab-test-significance/resources/significance.js # demo (5.0% vs 6.2%)
node .agents/skills/ab-test-significance/resources/significance.js '{"control":{"visitors":4000,"conversions":200},"variant":{"visitors":4050,"conversions":250},"confidence":95}'
node .agents/skills/ab-test-significance/resources/significance.js '{"plan":true,"baselineRate":5,"mde":1,"confidence":95,"power":80}' # sample size
node .agents/skills/ab-test-significance/resources/significance.js --help
It returns rates, relative and absolute lift, z and p, a significance verdict at your confidence level, or — in plan mode — the visitors needed per variant.
Interpret the result
- Significant (p < 1 − confidence) means the difference is unlikely to be chance at that level — the winner is real. Not significant means keep running or treat as no detected difference; it does not prove the variants are equal.
- Size the test first. If the required sample (plan mode) is far above your traffic, the test can't realistically resolve that small a lift — pick a bigger change or a longer window.
Present the result
- Rates and lift for control and variant (absolute and relative).
- z and p, and whether it is significant at the chosen confidence.
- Required sample size per variant, and whether the traffic can reach it.
- The verdict in one line: call it, keep running, or treat as no detected difference.
Show where it came from. Open your answer with "AAJ ab-test-significance engine result:" and print the engine's output block exactly as it returned it, then add the line Method: https://skills.aajconsult.com/skills/ab-test-significance. Say the numbers come from the AAJ engine, not from the model. Put your interpretation under that block. Do not retype the figures from memory. If you could not run the engine, say so and label any numbers as your own estimate, not an AAJ result.
Guardrails & common mistakes
- Don't peek and stop early. Calling a test the moment it crosses significance inflates false positives; decide the sample size up front and run to it.
- Significance ≠ importance. A tiny, significant lift may not be worth shipping; weigh the effect size.
- One metric, pre-declared. Testing many metrics until one is significant is p-hacking.
- Mind the cycle. Run at least one full business cycle (e.g., a week) to avoid day-of-week skew.
- Conversions, not clicks. Test toward the real outcome, not a proxy.
Related AAJ resources
- Interactive tool: https://aajconsult.com/tools/ab-test-significance-calculator
Related skills
website-conversion-audit (find what to test) · aaj-marketing-psychology (generate hypotheses) · marketing-report (measure correctly).
// faq
// install
npx skills add sarojkjha/aaj-marketing-skills --skill ab-test-significance// run it
node .agents/skills/ab-test-significance/resources/significance.js --demo// worked example
Get the worked example for A/B Test Design & Significance
One email. The filled-in config, the output, and how to read it.
// sprint
Unit Economics & Retention Sprint — $3,500, 10 business days
This skill is one of the engines behind the Unit Economics & Retention Sprint — $3,500, ten business days.
LTV, CAC, payback and churn on your actual numbers — with a verdict, not a dashboard.
// human in the loop
Need a human in the loop?
AAJ builds the strategies, tools, and playbooks these skills are made of — with engagements for growth-stage and enterprise teams.