CHAPTER 26
DEEP DIVE · FROM BASICS TO CODE

Understanding: Product metrics and trustworthy A/B tests

You can read this chapter before running the experiment. Then return to the live trace and match each concept to a real event.

FIRST, IN PLAIN LANGUAGE

Product metrics are an instrument panel, not a scoreboard. First state what valuable behavior should change, then record events and compare populations. A larger click count proves little by itself: ask who clicked, after which experience, whether they reached value, and whether another outcome became worse.

TECHNICAL FOUNDATION

Product analytics connects a decision question, a measurable event contract, a population, and an observation window. A funnel measures transitions between steps, a cohort groups users by a shared start, retention measures return to a valuable action, and a controlled A/B experiment uses random assignment to estimate causal impact. Decisions combine one primary metric with guardrails, quality checks, and an interval of plausible effects.

Why it mattersAn AI Product Engineer must not only ship a feature quickly but establish whether it helps. Faulty telemetry optimizes noise; missing guardrails improve one surface at the cost of errors or churn; invalid experiments confuse seasonality and self-selection with code impact.
WHERE THE WORK RUNS
01PRODUCT QUESTIONhypothesis · decision
02EVENT CONTRACTevent · actor · properties
03ANALYTICS STOREevents · cohorts · exposure
04DECISIONprimary · guardrails · CI
01 · GLOSSARY

Terms used in this experiment

Understand the words first, then the execution order.

01

A/B test

A controlled experiment that randomly and stably assigns units to variants and compares predefined outcomes.

02

North Star Metric

One leading measure of value received by users that connects to the long-term product outcome. It is not automatically revenue, DAU, or clicks.

03

Guardrail metric

A protective measure that must not deteriorate beyond an acceptable bound: error rate, latency, refunds, complaints, unsubscribe, or churn.

04

Event / property

An event records that an action happened at a time, such as report_generated. Properties describe context: userId, plan, locale, experiment variant, and duration.

05

Funnel

An ordered sequence of actions and the share of a population that reaches each next step within a defined time window.

06

Cohort

A group sharing a condition or start time, such as users registered in one week or first-time users of AI search.

07

Activation

The first verified moment a new user receives core value, not merely opening a page or registering.

08

Conversion

The share of an eligible population completing the target action: converters / eligible participants. The denominator is part of the definition.

09

Retention

The share of a starting cohort returning to a meaningful action later. Calendar, rolling, and bracket retention answer different questions.

10

Churn

Customers, subscriptions, or recurring revenue lost during a period. User churn and revenue churn must not be mixed.

11

Exposure event

Evidence that an experiment unit actually received a variant. Assignment without a display may not belong in triggered analysis.

12

Randomization unit

The entity independently assigned to a variant: user, account, device, session, or organization. Choose it from the feature interaction model.

13

MDE, sample size, and power

MDE is the smallest effect worth detecting. Smaller effects and noisier metrics need more sample. Power is the chance of detecting a real effect of the chosen size.

14

Confidence interval (CI)

A range of effect estimates compatible with the observations and model at a selected level. It communicates uncertainty better than one p-value.

15

Statistical significance

Evidence that the observed difference is difficult to explain under a null model, given assumptions. It does not prove importance, absence of bias, or the business hypothesis.

16

SRM

Sample Ratio Mismatch: observed variant allocation is implausibly different from plan, often indicating an assignment, exposure, or data bug.

02 · MECHANICS

What happens step by step

Each step maps to an observable runtime state.

  1. 01
    Start with a decision, not a dashboard

    Write what the team will do if the metric rises, stays inconclusive, or deteriorates.

  2. 02
    State the hypothesis

    Name the population, change, expected behavior, causal mechanism, and observation window.

  3. 03
    Choose a primary metric and guardrails

    One primary metric answers the main question; guardrails bound the acceptable cost of improving it.

  4. 04
    Specify the event contract

    Define name, business meaning, send moment, actor identity, required properties, owner, schema version, and deduplication rule.

  5. 05
    Validate identity and eligibility

    Decide how anonymousId merges into userId, who enters the population, and how bots, staff, and test accounts are excluded.

  6. 06
    Measure a baseline

    Before changing anything, inspect volume, variance, seasonality, missing or duplicate events, and definition stability.

  7. 07
    Choose the randomization unit

    Assign at a level where participants do not transfer treatment to each other; a B2B feature often needs account rather than user.

  8. 08
    Plan MDE, sample, and duration

    Predefine a useful effect, baseline rate, power, and error threshold. A short, small test does not become reliable because its uplift looks large.

  9. 09
    Record exposure once

    Persist assignment and log actual display before outcomes; avoid moving a unit between variants across requests or devices without an explicit design.

  10. 10
    Do not decide on every refresh

    Follow the chosen stopping rule. Repeated unplanned peeking increases the probability of a chance win.

  11. 11
    Check quality before effect

    Investigate SRM, event loss, duplicates, pre-treatment differences, broken guardrails, and inconsistent feature versions.

  12. 12
    Interpret magnitude and uncertainty

    Compare estimate and CI with MDE, inspect only planned segments, account for novelty, and monitor after rollout.

03 · CONTEXT

Where the result needs context

These details explain why similar code can sometimes produce a different trace.

01

A North Star cannot stand alone

One aggregate hides quality, distribution, and harm. Pair it with input metrics, guardrails, and cuts across important populations.

02

An event should name a business fact

button_clicked is tied to UI, while report_exported describes durable intent. Business semantics survive redesigns better than selectors.

03

The denominator defines the metric

Ten purchases among 100 exposed users and ten among 20 visitors are different conversion rates. Keep eligible non-converters with LEFT JOIN.

04

Retention needs a meaningful return event

Opening the app may be accidental. For a publishing editor, another save or publication may better represent value.

05

Calendar and rolling retention differ

Day-7 calendar asks about a specific period; rolling asks whether the user returned on day seven or later. Never compare them without definitions.

06

Assignment is not exposure

A user can receive a stored variant without opening the screen. Triggered analysis focuses on affected units, but its trigger must be defined before results.

07

The influence model determines randomization

If colleagues share a workspace document, per-user randomization contaminates groups. Assign the whole workspace together.

08

A CI is more useful than a winner label

An interval may allow both meaningful benefit and harm. Not significant means insufficient evidence, not a proven zero effect.

09

Peeking changes false-positive frequency

A fixed-horizon test assumes one planned analysis. Continuous monitoring needs a sequential method or a strict stopping rule.

10

Multiple testing needs control

Testing 20 equal metrics and many segments makes a chance winner more likely. Declare primary metric and planned slices in advance.

11

Novelty can be temporary

Users may explore a new interface or initially resist it. Cover a full business cycle and keep monitoring after rollout.

12

Practical and statistical significance differ

With huge samples a microscopic effect may be statistically significant while failing to pay for engineering, latency, or support.

01
Theory

First understand which parts of Node participate in execution.

02
Simplified code

Then remove instrumentation and focus on the central mechanism.

03
Runtime code

Finally match the model to the code that produces the live trace.

04 · Simplified code

A minimal model without instrumentation

NestJS + PostgreSQL · learning fragmentTypeScript + SQL
type ProductEventName =
  | 'project_created'
  | 'project_published';

@Injectable()
export class ProductMetricsService {
  constructor(private readonly db: Database) {}

  capture(input: {
    eventId: string;
    actorId: string;
    name: ProductEventName;
  }) {
    return this.db.query(
      'INSERT INTO product_events ' +
      '(event_id, user_id, event_name) ' +
      'VALUES ($1, $2, $3) ' +
      'ON CONFLICT (event_id) DO NOTHING',
      [input.eventId, input.actorId, input.name],
    );
  }
}
05 · Runtime code

The complete code executed by the scenario

This is not an alternative example: these are the functions and files used by the Run button.

ACTUAL SOURCE

The source is generated from the real server function. Scenarios using a child process or Worker include every participating file.

src/product-metrics-lab.js
scenario116 lines
function percentage(value) {
  return Number((value * 100).toFixed(2));
}

export function funnel(events, orderedSteps) {
  const progressByUser = new Map();

  for (const event of events) {
    const progress = progressByUser.get(event.userId) ?? 0;
    if (event.name === orderedSteps[progress]) {
      progressByUser.set(event.userId, progress + 1);
    }
  }

  return orderedSteps.map((step, index) => {
    const users = [...progressByUser.values()].filter(
      (progress) => progress > index,
    ).length;

    return { step, users };
  });
}

export function retention(users, day) {
  const eligible = users.filter((user) => user.observedDays >= day);
  const returned = eligible.filter((user) => user.activeDays.includes(day));

  return {
    eligible: eligible.length,
    returned: returned.length,
    rate: eligible.length ? returned.length / eligible.length : 0,
  };
}

export function compareBinaryExperiment(control, variant) {
  const controlRate = control.conversions / control.users;
  const variantRate = variant.conversions / variant.users;
  const absoluteLift = variantRate - controlRate;
  const standardError = Math.sqrt(
    (controlRate * (1 - controlRate)) / control.users +
      (variantRate * (1 - variantRate)) / variant.users,
  );
  const margin95 = 1.96 * standardError;
  const total = control.users + variant.users;

  return {
    controlRate,
    variantRate,
    absoluteLift,
    confidenceInterval95: [absoluteLift - margin95, absoluteLift + margin95],
    allocation: {
      control: control.users / total,
      variant: variant.users / total,
    },
  };
}

export async function productMetricsExperiment(emit) {
  const events = [
    { userId: 'u1', name: 'signup_completed' },
    { userId: 'u1', name: 'project_created' },
    { userId: 'u1', name: 'project_published' },
    { userId: 'u2', name: 'signup_completed' },
    { userId: 'u2', name: 'project_created' },
    { userId: 'u3', name: 'signup_completed' },
  ];
  const users = [
    { observedDays: 14, activeDays: [0, 1, 7], variant: 'control' },
    { observedDays: 14, activeDays: [0, 3], variant: 'control' },
    { observedDays: 14, activeDays: [0, 7], variant: 'variant' },
    { observedDays: 3, activeDays: [0, 1], variant: 'variant' },
  ];

  emit(
    'instrumentation',
    'contract',
    'События используют стабильные имена, userId и server timestamp',
  );

  const funnelResult = funnel(events, [
    'signup_completed',
    'project_created',
    'project_published',
  ]);
  emit(
    'funnel',
    'result',
    funnelResult.map(({ step, users }) => `${step}=${users}`).join(' → '),
  );

  const day7 = retention(users, 7);
  emit(
    'cohort',
    'result',
    `D7 retention=${percentage(day7.rate)}%; denominator=${day7.eligible}`,
  );

  const experiment = compareBinaryExperiment(
    { users: 5_000, conversions: 1_000 },
    { users: 5_100, conversions: 1_096 },
  );
  const [low, high] = experiment.confidenceInterval95.map(percentage);
  emit(
    'experiment',
    'result',
    `Control=${percentage(experiment.controlRate)}%; variant=${percentage(experiment.variantRate)}%; lift CI95=[${low}, ${high}] п.п.`,
  );
  emit(
    'decision',
    'guardrail',
    'Решение требует заранее выбранной primary metric, guardrails и проверки SRM',
  );

  return { funnel: funnelResult, retention: day7, experiment };
}

The application instruments its own live trace: rows and timestamps are recorded by real emit(...) calls, while the scenario supplies source and lane labels. This is not a V8/libuv profiler or a direct view of their internal queues. await and Promise keep the HTTP stream open until the scenario completes.

06 · RECIPES

Practical patterns worth keeping nearby

Compare the goal, code, and caveats instead of memorizing syntax without a model.

01

Nest: a server event contract

Prevent a client from impersonating the actor or submitting an unknown event name.

const allowedNames = [
  'report_created',
  'report_published',
] as const;

type ProductEventName = typeof allowedNames[number];

@Injectable()
export class ProductMetricsService {
  capture(
    actorId: string,
    name: ProductEventName,
    properties: Record<string, string | number>,
  ) {
    return this.events.insert({
      eventId: randomUUID(),
      actorId,
      name,
      properties,
      schemaVersion: 1,
    });
  }
}
  • as const preserves string literals instead of widening to string.
  • ProductEventName becomes a union of allowed names.
  • Actor arrives separately from authenticated context.
  • A compile-time type does not replace runtime validation of external JSON.
02

SQL: signup funnel

Count users reaching signup → project → publish within seven days.

WITH first_steps AS (
  SELECT user_id,
    min(occurred_at) FILTER (
      WHERE event_name = 'signup_completed'
    ) AS signed_up_at,
    min(occurred_at) FILTER (
      WHERE event_name = 'project_created'
    ) AS project_at,
    min(occurred_at) FILTER (
      WHERE event_name = 'project_published'
    ) AS published_at
  FROM product_events
  GROUP BY user_id
)
SELECT
  count(*) FILTER (WHERE signed_up_at IS NOT NULL) AS signup,
  count(*) FILTER (WHERE project_at >= signed_up_at) AS project,
  count(*) FILTER (
    WHERE published_at >= project_at
      AND published_at < signed_up_at + interval '7 days'
  ) AS published
FROM first_steps;
  • WITH creates the intermediate first_steps relation.
  • min finds each user's first event of a type.
  • FILTER limits rows entering one aggregate.
  • Conditions preserve funnel order and its time window.
03

SQL: weekly retention

Measure return to publishing during the week after activation.

WITH activated AS (
  SELECT user_id,
         date_trunc('week', min(occurred_at)) AS cohort_week
  FROM product_events
  WHERE event_name = 'project_published'
  GROUP BY user_id
), returned AS (
  SELECT DISTINCT a.user_id, a.cohort_week
  FROM activated a
  JOIN product_events e ON e.user_id = a.user_id
   AND e.event_name = 'project_published'
   AND e.occurred_at >= a.cohort_week + interval '1 week'
   AND e.occurred_at <  a.cohort_week + interval '2 weeks'
)
SELECT a.cohort_week,
       count(*) AS activated_users,
       count(r.user_id) AS retained_users,
       count(r.user_id)::numeric / count(*) AS week_1_retention
FROM activated a
LEFT JOIN returned r USING (user_id, cohort_week)
GROUP BY a.cohort_week
ORDER BY a.cohort_week;
  • A user joins a cohort at their first publication.
  • DISTINCT prevents frequent publishers from inflating the numerator.
  • LEFT JOIN keeps non-retained users in the denominator.
  • This definition is calendar week-1, not rolling retention.
04

SQL: subscription churn

Avoid dividing lost subscriptions by the entire historical customer base.

SELECT
  date_trunc('month', cancelled_at) AS month,
  count(*) AS cancelled,
  count(*)::numeric /
    nullif(max(active_at_month_start), 0) AS user_churn
FROM subscription_cancellations
GROUP BY date_trunc('month', cancelled_at)
ORDER BY month;
  • The denominator is active subscriptions at the start of that period.
  • nullif prevents division by zero.
  • Revenue churn sums lost recurring revenue instead of users.
05

Stable variant assignment

Give one user one variant without a central random generator.

function assignVariant(
  experimentKey: string,
  userId: string,
) {
  const bucket = createHash('sha256')
    .update(experimentKey + ':' + userId)
    .digest()
    .readUInt32BE(0) % 10_000;

  return bucket < 5_000 ? 'control' : 'treatment';
}
  • Experiment key separates independent tests.
  • A deterministic hash repeats assignment on another Nest instance.
  • A production implementation versions allocation and stores exposure.
  • Use accountId for an account-level feature.
06

SQL: check SRM before reading outcomes

Compare observed variant sizes with the planned allocation.

SELECT variant, count(DISTINCT user_id) AS participants
FROM experiment_exposures
WHERE experiment_key = $1
GROUP BY variant;

-- A 50/50 plan expects similar counts.
-- The experiment service runs a statistical SRM check
-- before evaluating the primary metric.
  • DISTINCT matches the user randomization unit.
  • A material mismatch needs investigation, not manual result correction.
  • The threshold depends on sample and allocation; visual inspection alone is insufficient.
07

Write the experiment card before launch

Make decision criteria auditable before a result is visible.

const checkoutExperiment = {
  hypothesis:
    'Shorter form increases completed purchases',
  unit: 'user',
  population: 'authenticated users entering checkout',
  primaryMetric: 'purchase within 24h of exposure',
  guardrails: ['payment_error_rate', 'refund_rate'],
  allocation: { control: 0.5, treatment: 0.5 },
  mde: 0.02,
  stoppingRule: 'planned sample and full business weeks',
};
  • Clarify whether MDE 0.02 means absolute percentage points or relative uplift.
  • Guardrails need acceptable limits, not names alone.
  • A frozen plan reduces freedom to select a convenient post-hoc result.
07 · PRODUCTION CASES

How a learning mistake becomes an incident

A realistic service: the original code, observable failure, corrected implementation, and why the correction works.

CASE 01

Activation rises because an event is recorded before business success

The team calls a user activated after their first project is created. The controller sends an unawaited event before the project write finishes, and every HTTP retry creates another event.

INCIDENT CONTEXT

A database failure still leaves false activation behind, while a retry doubles the events. The dashboard reports growth absent from business data and can drive the wrong product decision.

BEFOREPROBLEMATIC IMPLEMENTATION
@Post()
async create(
  @CurrentUser() user: User,
  @Body() dto: CreateProjectDto,
) {
  this.metrics.capture({
    eventId: randomUUID(),
    name: 'project_created',
    userId: user.id,
  }); // Promise is not awaited

  return this.projects.create(user.id, dto);
}

The event is not coupled to the project commit, and a new randomUUID on every retry cannot identify a duplicate. The method may fail after analytics has already accepted a success.

AFTERCORRECTED IMPLEMENTATION
@Injectable()
export class ProjectsService {
  constructor(private readonly db: Database) {}

  create(userId: string, dto: CreateProjectDto) {
    return this.db.transaction(async (tx) => {
      const project = await tx.one(
        'INSERT INTO projects (user_id, name) ' +
        'VALUES ($1, $2) RETURNING id, name',
        [userId, dto.name],
      );

      await tx.query(
        'INSERT INTO product_event_outbox ' +
        '(dedupe_key, event_name, user_id, properties) ' +
        'VALUES ($1, $2, $3, $4::jsonb) ' +
        'ON CONFLICT (dedupe_key) DO NOTHING',
        [
          'project_created:' + project.id,
          'project_created',
          userId,
          JSON.stringify({ projectId: project.id }),
        ],
      );

      return project;
    });
  }
}

The project and outbox event commit in one database transaction. A stable dedupe_key makes retries safe, and a publisher delivers only committed events to analytics.

FUNCTIONS AND CONSTRUCTS

What the unfamiliar calls from both code samples actually do.

@Injectable()
A Nest decorator that marks the class as a provider the IoC container can construct and inject into a controller.
db.transaction(callback)
Opens a transaction: either both the project and outbox event persist, or both changes roll back.
tx.one(sql, values)
An illustrative repository helper that runs parameterized SQL and requires exactly one returned row.
$1, $2 and values
PostgreSQL placeholders. Values travel separately from SQL text and positions start at one.
RETURNING
Returns fields from the row just written without a second SELECT; the event needs the new project id.
ON CONFLICT ... DO NOTHING
A retry with the same unique dedupe_key does not create a second event row.
outbox
A table of event-delivery intents. A separate worker reads committed rows and reliably forwards them to analytics.
WHY THE CORRECTION WORKS

A product metric begins with a correct business event. Delivery must survive failures, retries, and concurrency; statistical precision is useless on top of incorrect facts.

WHAT PRODUCTION SHOWED
  • project_created events outnumber real first projects.
  • Activation moves with the HTTP retry rate.
  • Events reference projectIds absent from the primary database.
CASE 02

An A/B test “wins” because every request gets a new random variant

A new checkout form is selected with Math.random() on every page load. Analysis groups only completed orders by the last variant the user saw.

INCIDENT CONTEXT

One user enters both groups, exposure is duplicated, and non-buyers are absent from the denominator. The observed difference has no valid causal interpretation.

BEFOREPROBLEMATIC IMPLEMENTATION
@Get('checkout')
async checkout(@CurrentUser() user: User) {
  const variant = Math.random() < 0.5 ? 'A' : 'B';

  await this.metrics.capture({
    name: 'checkout_exposed',
    userId: user.id,
    properties: { variant },
  });

  return this.checkoutPage.render({ variant });
}

// Analysis bug: buyers only
SELECT variant, count(*)
FROM orders
GROUP BY variant;

Assignment is unstable and violates the randomization unit. The query counts outcomes without all exposed participants, so it does not compute a conversion rate.

AFTERCORRECTED IMPLEMENTATION
@Injectable()
export class CheckoutExperiment {
  constructor(private readonly db: Database) {}

  async getVariant(userId: string) {
    const bucket = createHash('sha256')
      .update('checkout-v2:' + userId)
      .digest()
      .readUInt32BE(0) % 10_000;

    const variant = bucket < 5_000
      ? 'control'
      : 'treatment';

    await this.db.query(
      'INSERT INTO experiment_exposures ' +
      '(experiment_key, user_id, variant, exposed_at) ' +
      'VALUES ($1, $2, $3, now()) ' +
      'ON CONFLICT (experiment_key, user_id) DO NOTHING',
      ['checkout-v2', userId, variant],
    );

    return variant;
  }
}

WITH exposed AS (
  SELECT user_id, variant, min(exposed_at) AS exposed_at
  FROM experiment_exposures
  WHERE experiment_key = 'checkout-v2'
  GROUP BY user_id, variant
), converted AS (
  SELECT DISTINCT e.user_id
  FROM exposed e
  JOIN orders o ON o.user_id = e.user_id
   AND o.created_at >= e.exposed_at
)
SELECT e.variant,
       count(*) AS participants,
       count(c.user_id) AS conversions,
       count(c.user_id)::numeric / count(*) AS rate
FROM exposed e
LEFT JOIN converted c USING (user_id)
GROUP BY e.variant;

The hash keeps each user in one group. A unique exposure provides an honest denominator, LEFT JOIN retains non-buyers, and only outcomes after first exposure count.

FUNCTIONS AND CONSTRUCTS

What the unfamiliar calls from both code samples actually do.

createHash(...).update(...)
Deterministically turns experiment key plus userId into bytes, so the same user receives the same bucket.
readUInt32BE(0) % 10_000
Reads a number from the first four hash bytes and maps it to bucket 0–9999 for percentage rollout.
UNIQUE (experiment_key, user_id)
A database constraint prevents two assignments for the same user in one experiment.
WITH exposed AS (...)
A CTE names an intermediate relation and makes the analysis denominator explicit.
LEFT JOIN
Retains every exposed user even with no matching order; a missing order is a non-conversion.
count(c.user_id)::numeric / count(*)
Divides converters by all group participants. ::numeric prevents integer division.
WHY THE CORRECTION WORKS

An A/B test needs a stable randomization unit, a single assignment, valid exposure, and a predefined outcome window. Then SRM, confidence intervals, guardrails, and practical effect size still need inspection.

WHAT PRODUCTION SHOWED
  • The same user_id appears in control and treatment.
  • Group proportions materially differ from the planned 50/50 split.
  • Conversion uses orders rather than exposed users as its denominator.
  • The effect disappears after the first days because of novelty.
08 · DO NOT CONFUSE

Common misconceptions

The myth is on the left; the accurate model is on the right.

MYTH

More events always mean more understanding.

ACTUALLY

Without an owner, schema, and product question, events create contradictory definitions and storage cost.

MYTH

DAU always works as a North Star.

ACTUALLY

DAU measures presence, not necessarily received value; the metric must represent this product's core outcome.

MYTH

Retention is any repeat visit.

ACTUALLY

It requires an explicit cohort, return action, and time window.

MYTH

If p < 0.05, the feature is definitely useful.

ACTUALLY

Inspect design, SRM, data quality, CI, effect size, guardrails, and implementation cost.

MYTH

No significance means the variants are equal.

ACTUALLY

The test may lack power; the CI shows which effects remain compatible with the data.

MYTH

Math.random() on every request creates an A/B test.

ACTUALLY

A unit needs stable assignment, and exposure plus outcome must link back to it correctly.

MYTH

Stop as soon as the dashboard turns green.

ACTUALLY

Unplanned peeking and stopping increase false-positive decisions.

MYTH

A segment discovered after the test already proves an effect.

ACTUALLY

A post-hoc segment is a new hypothesis that needs confirmation on independent data.

09 · SELF-CHECK

Explain it in your own words

If you can explain the answer without quoting documentation, your mental model is starting to take shape.

  1. Why does a North Star need guardrail metrics?
  2. How does an event differ from its properties?
  3. What belongs in the denominator of a conversion rate?
  4. How do cohort and return event define retention?
  5. How do user churn and revenue churn differ?
  6. Why is assignment not always exposure?
  7. How do you choose user versus account as a randomization unit?
  8. How do MDE, sample size, and power relate?
  9. What does a confidence interval add beyond a significant label?
  10. Why do peeking and multiple testing increase chance wins?
  11. What does SRM signal, and why should it be checked before outcomes?
  12. How does a transactional outbox protect product events?
  13. Why does a successful A/B test not remove post-rollout monitoring?