Incrementality And Mmm

Incrementality & MMM

Answer the question attribution cannot: would this conversion have happened anyway?

Attribution models tell you which touchpoint preceded a conversion. They cannot tell you which touchpoint caused it, because they only observe the world where the ad ran. The gap between those two things is largest exactly where reported performance looks best — retargeting and branded search routinely show excellent attributed ROAS and much weaker incremental ROAS, because they intercept people who were already going to convert.

The only way to close that gap is to withhold the spend from a comparable group and measure the difference. That's uncomfortable — you're deliberately forgoing revenue to learn something — which is why most teams never do it, and why the ones that do usually reallocate budget afterwards.

The failure mode this skill exists to prevent is the underpowered test. A holdout that finds "no significant difference" when it was never large enough to detect a meaningful effect is not evidence the channel doesn't work. It gets reported as though it were, and budget moves on the strength of it.

When to use

The user is designing or reading a holdout, geo, or lift test — or questioning whether attributed conversions are real. For on-site A/B tests of a page or flow, use ab-test-significance; the statistics overlap but the design question is different. For reporting results upward, use marketing-report.

Before you start

1. Read the brand/product context (.agents/product-marketing.md) for the business model and conversion definition.
2. Fix the conversion definition first. Incrementality on a soft metric tells you little. Measure the thing with revenue attached.
3. Get the real baseline rate, not a rounded estimate — sample size is highly sensitive to it.
4. Decide the lift worth detecting before running anything. This is a business question, not a statistical one: below what lift would you not change the budget?
5. Check you can actually hold out. Some channels can't be split cleanly. Geo tests exist for exactly that case.

Method

Design before you spend. Sample size follows from four inputs: baseline rate, the lift you care about, confidence, and power. Small lifts on low baselines need very large samples — a 5% lift on a 2% conversion rate needs hundreds of thousands per group, which for most Seed–Series B companies means the test is impossible and the honest answer is to test a bigger intervention.

Then hold everything else constant. A test contaminated by a pricing change, a competing campaign, or a seasonal shift measures nothing. Duration matters here: run at least one full weekly cycle, and be suspicious of tests running past six weeks, where drift becomes near-certain.

Then read it honestly. Three outcomes, not two:

• Incremental — significant lift. The spend caused conversions.
• Not significant, adequately powered — a real finding. The channel's attributed conversions are suspect.
• Inconclusive — the test could never have detected the effect you'd care about. This is not evidence of anything.

That third category is the one that gets misreported, and the engine calls it explicitly.

See resources/incrementality-guide.md for holdout vs geo test design, what to expect by channel, the MMM-lite section, and a worked example.

Run the engine

Design a test:

Paths assume you installed with npx skills add. From a clone of this repo, use skills/incrementality-and-mmm/resources/… instead.

node .agents/skills/incrementality-and-mmm/resources/incrementality.js  # demo
node .agents/skills/incrementality-and-mmm/resources/incrementality.js --baseline 2.5 --mde 15 --daily 4000 --value 400
node .agents/skills/incrementality-and-mmm/resources/incrementality.js --baseline 1.2 --mde 25 --power 0.90 --daily 20000

Read one out — supply --treat-conv and it switches mode:

node .agents/skills/incrementality-and-mmm/resources/incrementality.js --control-n 50000 --control-conv 1200 \
     --treat-n 50000 --treat-conv 1380 --spend 15000 --value 400
node .agents/skills/incrementality-and-mmm/resources/incrementality.js --json --control-n 5000 --control-conv 120 \
     --treat-n 5000 --treat-conv 132
node .agents/skills/incrementality-and-mmm/resources/incrementality.js --help

Design mode returns sample per group, duration, and the value you forgo by holding out — which is the number that decides whether the test is worth running. Readout mode returns lift, p-value, confidence interval, incremental conversions against the counterfactual, iROAS, incremental CAC, and the smallest lift the test could actually detect.

Workflow

1. State the decision the test will inform. If no budget decision hangs on the result, don't run it.
2. Set the lift worth detecting from that decision, not from what seems achievable.
3. Run design mode. If the duration exceeds about six weeks, the test as specified isn't viable — either raise the detectable lift, use a geo test, or test a larger intervention.
4. Weigh the holdout cost against the spend at stake. Withholding $300K to validate a $50K channel is the wrong trade.
5. Freeze everything else for the window. Document what else was running so contamination is visible later.
6. Run to the planned duration. Stopping when the result looks good inflates false positives — this is the single most common way lift tests mislead.
7. Run readout mode. Report the verdict, not just the p-value.
8. Act on iROAS, not attributed ROAS. Where they diverge sharply, trust the test.

Present the result

Lead with the verdict and iROAS — those drive the budget decision. Then the incremental conversions against the counterfactual, since that's the number people can reason about. Then significance and the detectable-lift caveat.

If the test was inconclusive, say so plainly and give the sample size that would have answered it. Do not let an inconclusive test be reported as a negative finding.

Guardrails & common mistakes

• Never report an underpowered null as "the channel doesn't work." It means the test was too small. The engine separates these deliberately.
• Don't stop early. Peeking and stopping on a favourable result badly inflates false positives. Fix the duration in advance and honour it.
• Attributed ROAS and iROAS are different numbers. Expect them to diverge most on retargeting and branded search, where attribution is most generous.
• Geo tests need comparable geos. Match on baseline conversion rate and volume, not just population. Unmatched geos produce differences that have nothing to do with the campaign.
• One change at a time. A test that alters spend and creative and landing page tells you something happened, not what.
• Results decay. An incrementality finding is a snapshot of a market, a creative, and a season. Re-test periodically rather than treating one result as permanent.
• Don't hold out so much that you damage the business. The design mode prices this; use it. A statistically ideal test that costs a quarter's pipeline is not a good test.
• MMM and incrementality answer different questions. MMM allocates across channels using historical data and is prone to correlation artefacts; incrementality establishes causation for one channel. Where they disagree, the experiment wins — and the honest use of MMM is to generate hypotheses that incrementality then tests.

Related AAJ resources

• Interactive tool: https://aajconsult.com/tools/ab-test-significance-calculator
• Interactive tool: https://aajconsult.com/tools/marketing-kpi-report

Related skills

ab-test-significance (on-site experiments — same statistics, different design question) · paid-media-budget-allocation (where the reallocation decision lands) · unit-economics (the value-per-conversion input this depends on) · marketing-report (reporting the result upward) · marketing-attribution (the model this is designed to check).