- Define the comparison you want to learn from
- Save assignment before exposure
- Set the outcome window and decision in advance
- Use the calculator for reconciled aggregate totals
- Audit execution before interpreting the difference
- Report uncertainty and keep the original test record
- Read vendor reports with their attribution rules attached
- Bring one measurable texting question to Blackleaf
- Frequently asked questions
Plan a dispensary SMS holdout test before the campaign sends. Randomly assign eligible customers to the campaign or a holdout from that specific optional marketing intervention, save the assignment, and compare outcomes over the same fixed period. Keep failed and undelivered assignments in the primary comparison.
This guide provides a pre-send test plan, an execution checklist and a calculator for aggregate outcomes. The broader SMS ROI guide explains accounting and attribution. The one-store pilot guide covers the wider operating rollout.
Define the comparison you want to learn from
Write one question: for example, whether one optional win-back campaign changes completed-purchase behavior and contribution among a defined eligible audience. Specify what the campaign group receives and what the holdout does not receive. A test of a message plus a discount estimates that package; it does not isolate the wording alone.
Distinguish a single-campaign holdout from a program holdout. In the first, both groups may continue under the same background marketing rules except for the tested campaign. In the second, the holdout is excluded from a defined program for a stated period. Record the actual exclusions and keep necessary order, service and safety communication available.
Start with current permission and suppression, stable customer identities, supported source data and an appropriate message. A holdout design does not make an otherwise unsupported recipient, business, route or promotion eligible.
Save assignment before exposure
- Freeze eligibility. Save the eligible population, store scope, cutoff and pre-send exclusions. Resolve duplicate profiles and shared numbers before assignment where possible. Do not select a control group because it looks less likely to buy.
- Assign at the right unit. Use a documented random process and retain the resulting customer IDs and group labels. Keep each customer in one group across the stores included in the test. If households, stores or other clusters share exposure, plan assignment and analysis for that structure rather than treating every row as independent.
- Save the allocation. Keep the intended split, randomization method or reproducible assignment reference, creation time and final counts. Do not refresh a volatile spreadsheet random column after sending or reassign people based on their response.
- Check the campaign exclusion. Rehearse the saved lists and current suppression using approved records. Confirm holdout customers cannot enter the tested campaign through a duplicate list, manual send, API job or automation. New withdrawals still stop affected outreach; the historical assignment remains in the analysis record.
A statistically planned allocation is not necessarily an equal split. There is no universal “hold out 10%” rule. Required size depends on the baseline outcome, minimum effect worth detecting, desired precision and power, available audience and design. Ask an analyst to plan this before spending; a small audience can yield an inconclusive test.
Set the outcome window and decision in advance
| Plan item | Record the exact choice |
|---|---|
| Primary outcome | One defined measure, such as unique customers with a qualifying completed purchase or contribution per assigned customer. Separate purchasers from order count. |
| Window | Start and end timestamps/time zone for both groups, covered stores/channels, qualifying transactions, returns rule and a data-maturity/readout date. |
| Economics | Consistent net sales, discounts, returns, product and variable selling costs. List additional messaging, platform, creative and labor costs separately; count each once. |
| Other exposure | Background promotions, email, push, referrals and staff offers allowed in both groups; what counts as contamination and how it will be recorded. |
| Analysis and decision | Original assigned groups, planned uncertainty method, missing-data review, minimum worthwhile benefit and customer/operating guardrails. Specify the final readout and any planned interim checks. |
Choose a common time origin rather than starting the campaign group at successful delivery and giving the holdout an unrelated window. A campaign’s attribution window is a different reporting rule; the experiment needs outcomes for both assigned groups, including purchases not credited by the messaging dashboard.
Do not extend the window, remove a poor store or pick the best subgroup after seeing the results and present it as the original test. Label exploratory findings separately. Repeatedly checking for a positive result and stopping then needs an appropriate sequential design; ordinary repeated peeking can overstate evidence.
Use the calculator for reconciled aggregate totals
Enter the original assigned counts, unique qualifying purchasers and contribution totals for the same completed window. Include confirmed zero-purchase customers in their assigned denominator. Unknown or missing outcomes are not zeros. Resolve coverage or use a justified missing-data analysis before interpreting results.
Contribution here means net sales after discounts and returns, minus product and variable selling costs, before the additional campaign expenses entered below. Use the same accounting basis in both groups. Contributions may be negative. Enter aggregate totals only; customer records are not needed in this tool.
The calculator shows arithmetic point estimates. It does not randomize customers, configure exclusions, validate a test, forecast a rollout, or calculate statistical significance, confidence intervals or sample size. Positive results need the same design and uncertainty review as negative or zero results.
Formulas and a static example
Purchaser rate = unique qualifying purchasers ÷ originally assigned customers. The rate difference is campaign rate minus holdout rate. Multiplying that difference by the campaign assigned count gives an estimated additional-purchaser count at that group size; it can be fractional and is not an observed count of extra orders.
Contribution difference at campaign-group size = (campaign contribution ÷ campaign assigned − holdout contribution ÷ holdout assigned) × campaign assigned. Subtract additional campaign cost once for the net difference. Divide that net difference by campaign assigned for a per-customer figure.
Fictional arithmetic example, not a Blackleaf customer result: 1,000 assigned to each group; 120 campaign purchasers and 110 holdout purchasers; contribution of $6,000 and $5,500 before $150 additional campaign cost. Rates are 12% and 11%, a 1-percentage-point difference. The point estimates are 10 additional purchasers, $500 additional contribution before cost and $350 net difference after cost. These invented totals provide no uncertainty analysis or proof of an effect.
Audit execution before interpreting the difference
Keep assignment, exposure and outcomes as separate records. Trace campaign IDs, current suppression, send attempts, delivered/failed/unknown states, cross-channel promotions and qualifying transactions. Preserve refunds and costs using the planned rule. Do not require a link click or promo-code redemption for a purchase to count in the experiment if the primary outcome was all qualifying purchases.
- Assignment integrity: reconcile expected and actual group counts, duplicates, late additions and missing IDs. Investigate unexplained allocation imbalances rather than simply normalizing a potentially broken experiment.
- Contamination and interference: log holdout exposure through other lists, shared offers, household members or stores. Keep original assignment and disclose departures; do not selectively remove customers after observing purchases.
- Outcome coverage: compare missingness and match quality across groups. A POS outage, unmatched online orders or delayed returns can distort the comparison. Do not report unknown purchases as no purchases.
- Operational changes: record stockouts, price changes, closures and staffing changes. A test estimates the campaign as actually operated within its scope, not every future offer or store.
Use the field-mapping checklist for source identity and event meaning, and confirm actual integration fields and refresh timing before launch. An available customer sync does not establish full purchase, return or margin coverage.
Report uncertainty and keep the original test record
A small positive point estimate can occur by chance. Plan an appropriate uncertainty analysis for the actual randomization unit and outcome. Customer contribution can be highly variable; two group totals alone do not reveal that distribution. This calculator intentionally does not invent a profit interval from insufficient inputs.
If the plausible range includes material losses and useful gains, the result may be inconclusive. That is different from proving no effect. Review the prespecified decision, data quality and customer guardrails before expanding, stopping or planning a new test. Do not repeatedly rerun until one result looks favorable without accounting for the testing process.
The HM Treasury evaluation methods guide discusses randomized comparison, sample adequacy and control spillover. Google’s certainty guidance explains why point estimates and uncertainty differ. These are measurement references, not a claim that Blackleaf implements those platforms’ experiment features or confidence conventions.
Test ID / owner / plan saved at: Question and exact intervention: Eligible population / exclusions / stores: Randomization unit / method / assignment reference: Planned allocation / saved actual assigned counts: Holdout exclusions / permitted background communication: Primary outcome / common start and end / time zone: Returns rule / data maturity / readout date: Contribution definition / additional campaign costs: Sample-size and uncertainty plan / analyst: Minimum worthwhile result / customer guardrails: Contamination / assignment issues / missing outcomes: Actual results / uncertainty / evidence locations: Departures from plan and exploratory findings: Decision / limitations / next test:
Keep assignment and customer-level evidence in access-controlled records. The public example and calculator need only aggregate numbers. Preserve the original plan with departures and the final readout rather than replacing it with a cleaner-looking retrospective story.
Read vendor reports with their attribution rules attached
Official sources reviewed October 15, 2026. AIQ’s reporting guide distinguishes strict and broad attribution and documents window, engagement and export rules. Springbig’s brand platform describes campaign attribution analytics. Those reports can support operations; their labels alone do not establish what would have happened without the campaign.
Leafbuyer’s ROI page includes scenario assumptions and headline performance figures. A projected sales figure is different from a randomized incremental-contribution estimate. The legacy Happy Marketers page also describes sales attribution; its official ownership announcement says the marketing business is part of AIQ. Confirm the current account and reporting product instead of treating it as an independent current competitor.
Ask any provider for the assignment and outcome records, matching rules, included costs, time window and actual holdout/exclusion capability. Missing public documentation is not proof a feature is absent. Reported open rates, attributed purchases and customer testimonials are not substitutes for the comparison your test requires.
Bring one measurable texting question to Blackleaf
Blackleaf supports dispensary texting with campaigns, customer conversations and supported data connections. Confirm the audience exclusions, export access, customer identifiers, purchase/return coverage and costs needed for your test. This page does not announce an automatic randomization or statistical experiment-reporting feature.
Texting can complement supported existing POS systems without a POS switch. Blackleaf’s working POS is in production in California; production Metrc approval does not establish every jurisdiction or function. Connected POS AI agents are in production in California and are separate from this texting measurement plan. More states are rolling out soon.
For context, the source-qualified revenue and orders case study reports business trends without isolating texting’s contribution. A prospective holdout answers a different question and needs its own assignment, execution and outcome evidence.
Bring one eligible customer opportunity, your current source systems and the decision you need to make. Define the test before the send, then keep the result and its limits together.
Frequently asked questions
What is a dispensary SMS holdout test?
It is a planned comparison in which eligible customers are randomly assigned before sending to a campaign group or a group held out from that specific optional marketing intervention. Measure both groups over the same fixed window and retain the original assignment for the primary analysis.
Should failed or undelivered sends be removed from the campaign group?
No. Keep customers in their originally assigned group for the primary comparison. Report delivery and click diagnostics separately. Removing people based on what happened after assignment can bias the result.
How large should the holdout group be?
There is no universal percentage. Plan sample size and allocation around the baseline outcome, smallest effect worth detecting, desired power and uncertainty, costs and any clustered assignment. A small holdout can make estimates imprecise; this calculator does not perform that planning.
Can an A/B message test replace a no-campaign holdout?
An A/B test comparing two messages answers which version performs differently. If both groups receive the tested marketing intervention, it does not directly estimate the effect of sending that intervention versus withholding it.
Does a positive calculator result prove the campaign worked?
No. The calculator shows arithmetic point estimates from supplied totals. Interpretation needs valid assignment, consistent complete outcomes, contamination and missing-data review, and an appropriate uncertainty analysis.
Why use contribution per assigned customer?
It compares the groups on a common denominator while retaining zero-purchase customers and differences in repeat orders, basket value, discounts, returns and variable costs. Subtract additional campaign expense once, separately from costs already included in contribution.
Can the calculator produce a confidence interval for profit?
No. Group totals do not provide the distribution of individual customer contribution outcomes. Use an appropriate analysis of the underlying data and actual assignment design to quantify uncertainty.
What if a holdout customer receives another promotion?
Record the exposure, its timing and whether it conflicts with the predeclared comparison. Keep the original assignment and disclose contamination. Do not selectively remove customers after seeing their purchases to make the result cleaner.
Should essential order or safety messages be withheld?
No. Define the holdout around appropriate optional marketing. Keep necessary service, order and safety communication available; record how other permitted communication is handled in both groups.
Does a holdout test require Blackleaf POS?
No. Blackleaf texting can complement supported existing POS systems. Confirm stable customer IDs, purchase and return coverage, cost inputs and refresh timing before promising an outcome analysis.