Measure change. Quantify impact.

The core idea

Pre/post analysis - also called before/after analysis or interrupted time series - measures whether a metric changed between two consecutive time periods, typically separated by an intervention or event. The "pre" period captures baseline behavior; the "post" period captures behavior after the change.

Unlike a randomized controlled experiment, pre/post relies on temporal sequencing rather than randomization. You observe what happened before the change, observe what happened after, and apply statistical tests to decide whether the difference is larger than typical noise.

The output is a verdict - significant change or no detectable change - along with the magnitude (lift, absolute difference, effect size) and the uncertainty around that estimate (p-value, confidence interval).

Right tool, right job

Pre/post analysis is the right method when you cannot randomize traffic between treatment and control. Common scenarios:

• Site-wide redesigns - a new homepage shown to everyone
• Brand campaigns - TV, billboards, or PR that hits all visitors
• Infrastructure changes - CDN swaps, page-speed improvements
• Pricing changes - product price updates rolled out globally
• Policy or regulatory shifts - GDPR, cookie banners, etc.
• Seasonal events - Black Friday, flash sales, product launches

If you can split traffic between versions simultaneously, prefer A/B testing - it isolates the change from confounding factors. Use the A/B Test Calculator for randomized experiments.

Two-proportion Z-test

The Z-test compares conversion rates between the pre-period (rate p₁) and post-period (rate p₂):

Where p̂ is the pooled rate (combined conversions ÷ combined visitors) and Φ is the standard normal CDF. A p-value below your chosen alpha (typically 0.05) means the change is statistically significant.

Wilson score interval

For each individual rate, we compute the Wilson interval - more accurate than the standard Wald interval, especially for small samples or extreme rates:

Where z is the critical value for your confidence level (1.96 for 95%, 2.576 for 99%). The Wilson interval never produces nonsensical bounds outside [0, 1] and works reliably for proportions near 0% or 100%.

For the difference between rates, we use a Wald-based interval on the standard error of the difference.

Cohen's h

Effect size answers a critical question that p-values can't: is this change practically meaningful? With millions of users, a 0.01% change can be "statistically significant" but business-irrelevant.

The arcsine transformation makes h independent of sample size. Interpretation:

• |h| < 0.2 - Negligible effect
• 0.2 ≤ |h| < 0.5 - Small effect
• 0.5 ≤ |h| < 0.8 - Medium effect
• |h| ≥ 0.8 - Large effect

Relative vs absolute change

Both lift measures tell different stories:

A change from 2% to 3% is an absolute increase of 1 percentage point but a relative lift of 50%. Always specify which one you're reporting - they sound very different to stakeholders.

For revenue mode, we compute three additional metrics: revenue per visitor (monetization efficiency), revenue per conversion (average order value or deal size), and total revenue change.

Homepage redesign impact

An e-commerce store launches a new homepage targeting all visitors. The team compares 30 days before launch (15,000 visitors, 450 sales = 3% rate) to 30 days after (14,800 visitors, 520 sales = 3.5% rate).

Using this calculator: relative lift = +15.6%, p-value = 0.002, Cohen's h = 0.029 (negligible by Cohen's scale, but practically meaningful in revenue terms - likely +₹6.1L annual revenue at scale).

Verdict: Significant change observed. Caveat: launch coincided with end-of-season sale, so confounding is plausible. Plan an A/B test on a follow-up change to confirm causation.

Pricing page rewrite

A SaaS team rewrites pricing copy globally - no A/B test possible because of brand-consistency requirements. Pre-period: 8,200 trial signups, 412 paid conversions (5.0%). Post-period: 7,950 signups, 478 paid (6.0%).

Relative lift = +20.0%, absolute diff = +1.0pp, p = 0.012. Significant at 95% confidence with Cohen's h = 0.046.

Action: Pricing change appears to have driven measurable trial-to-paid improvement. Monitor the next 60 days for seasonal effects before committing the win to leadership.

Quote-to-policy flow change

An insurance comparison site updates its quote-to-policy checkout flow. Pre: 22,000 quotes, 1,540 policies (7.0%). Post: 21,500 quotes, 1,720 policies (8.0%). Revenue mode shows ATS (average ticket size) also rising from ₹4,800 to ₹5,200.

Relative lift in conversion = +14.3%, p < 0.001, total revenue +23.1%. Strong evidence of meaningful improvement.

Caveat: The team launched a parallel email campaign mid-window. They cannot fully separate the checkout-flow effect from the email effect - exactly the type of confounding pre/post can't resolve.

Feature-flag rollout (gradual)

A product team enables a new dashboard widget for all users on Monday. They pull pre-period data (Tue–Sun prior week) and post-period data (Tue–Sun current week) - same days-of-week, same length, comparable seasonality.

Conversion to a key downstream action: pre 12.4% → post 13.1%. p = 0.18. Not significant - could be random variation. Cohen's h = 0.021 (negligible).

Action: Either accept that the feature has no detectable effect, or collect more data over the next month. Consider a follow-up A/B test if you can fork the experience.

Comparing different seasons

Comparing November (post-launch) with September (pre-launch) is common but dangerous. Black Friday traffic, holiday intent, and weather all change conversion rates independent of your change.

Fix: Use year-over-year same-period (Nov 2025 vs Nov 2024), or rolling 30-day windows that fall in comparable seasonal positions. When in doubt, longer windows average out short-term noise.

Including launch-day anomalies

The first 24–72 hours after any change tend to be anomalous: cache misses, internal team testing, social-media spikes from announcement posts, partial rollouts. Including these days inflates noise.

Fix: Wait at least 3 days after launch before starting your post-period. Document the cutoff in your analysis notes.

Ignoring concurrent changes

Most teams ship multiple changes in the same week. If you launched a new pricing page and a new homepage simultaneously, pre/post can't tell you which one moved the needle - only that something did.

Fix: Maintain a launch log. When attributing change, list every concurrent change in the post-period. Be honest about confounding in your write-up.

Cherry-picking the post-period

If your change "wins" in the first two weeks but reverts to baseline by week four, declaring victory at week two is selection bias. Decide your post-period before looking at the data.

Fix: Pre-register your analysis window. If you said "30 days post," analyze 30 days - even if results look better at day 14.

Confusing significance with importance

With 1M+ users, almost any tiny change is "statistically significant." A 0.05% lift on a million users gives p < 0.001 but no real-world impact. Always check Cohen's h and absolute revenue change too.

Fix: Look at three things together: p-value (real or noise?), effect size (how big?), and business impact (worth shipping?). All three should support the decision.

Treating pre/post as causal proof

The most common LinkedIn post: "We launched X and saw +15%! Massive win!" Pre/post cannot prove X caused the lift. Maybe a competitor went down, maybe a viral moment happened, maybe seasonality shifted.

Fix: Frame results as "we observed +15% in the period after X" not "X caused +15%." For causal claims, run an A/B test on a follow-up change.