Product metrics are an instrument panel, not a scoreboard. First state what valuable behavior should change, then record events and compare populations. A larger click count proves little by itself: ask who clicked, after which experience, whether they reached value, and whether another outcome became worse.
Understanding: Product metrics and trustworthy A/B tests
You can read this chapter before running the experiment. Then return to the live trace and match each concept to a real event.
Product analytics connects a decision question, a measurable event contract, a population, and an observation window. A funnel measures transitions between steps, a cohort groups users by a shared start, retention measures return to a valuable action, and a controlled A/B experiment uses random assignment to estimate causal impact. Decisions combine one primary metric with guardrails, quality checks, and an interval of plausible effects.
Terms used in this experiment
Understand the words first, then the execution order.
A/B test
A controlled experiment that randomly and stably assigns units to variants and compares predefined outcomes.
North Star Metric
One leading measure of value received by users that connects to the long-term product outcome. It is not automatically revenue, DAU, or clicks.
Guardrail metric
A protective measure that must not deteriorate beyond an acceptable bound: error rate, latency, refunds, complaints, unsubscribe, or churn.
Event / property
An event records that an action happened at a time, such as report_generated. Properties describe context: userId, plan, locale, experiment variant, and duration.
Funnel
An ordered sequence of actions and the share of a population that reaches each next step within a defined time window.
Cohort
A group sharing a condition or start time, such as users registered in one week or first-time users of AI search.
Activation
The first verified moment a new user receives core value, not merely opening a page or registering.
Conversion
The share of an eligible population completing the target action: converters / eligible participants. The denominator is part of the definition.
Retention
The share of a starting cohort returning to a meaningful action later. Calendar, rolling, and bracket retention answer different questions.
Churn
Customers, subscriptions, or recurring revenue lost during a period. User churn and revenue churn must not be mixed.
Exposure event
Evidence that an experiment unit actually received a variant. Assignment without a display may not belong in triggered analysis.
Randomization unit
The entity independently assigned to a variant: user, account, device, session, or organization. Choose it from the feature interaction model.
MDE, sample size, and power
MDE is the smallest effect worth detecting. Smaller effects and noisier metrics need more sample. Power is the chance of detecting a real effect of the chosen size.
Confidence interval (CI)
A range of effect estimates compatible with the observations and model at a selected level. It communicates uncertainty better than one p-value.
Statistical significance
Evidence that the observed difference is difficult to explain under a null model, given assumptions. It does not prove importance, absence of bias, or the business hypothesis.
SRM
Sample Ratio Mismatch: observed variant allocation is implausibly different from plan, often indicating an assignment, exposure, or data bug.
What happens step by step
Each step maps to an observable runtime state.
- 01Start with a decision, not a dashboard
Write what the team will do if the metric rises, stays inconclusive, or deteriorates.
- 02State the hypothesis
Name the population, change, expected behavior, causal mechanism, and observation window.
- 03Choose a primary metric and guardrails
One primary metric answers the main question; guardrails bound the acceptable cost of improving it.
- 04Specify the event contract
Define name, business meaning, send moment, actor identity, required properties, owner, schema version, and deduplication rule.
- 05Validate identity and eligibility
Decide how anonymousId merges into userId, who enters the population, and how bots, staff, and test accounts are excluded.
- 06Measure a baseline
Before changing anything, inspect volume, variance, seasonality, missing or duplicate events, and definition stability.
- 07Choose the randomization unit
Assign at a level where participants do not transfer treatment to each other; a B2B feature often needs account rather than user.
- 08Plan MDE, sample, and duration
Predefine a useful effect, baseline rate, power, and error threshold. A short, small test does not become reliable because its uplift looks large.
- 09Record exposure once
Persist assignment and log actual display before outcomes; avoid moving a unit between variants across requests or devices without an explicit design.
- 10Do not decide on every refresh
Follow the chosen stopping rule. Repeated unplanned peeking increases the probability of a chance win.
- 11Check quality before effect
Investigate SRM, event loss, duplicates, pre-treatment differences, broken guardrails, and inconsistent feature versions.
- 12Interpret magnitude and uncertainty
Compare estimate and CI with MDE, inspect only planned segments, account for novelty, and monitor after rollout.
Where the result needs context
These details explain why similar code can sometimes produce a different trace.
A North Star cannot stand alone
One aggregate hides quality, distribution, and harm. Pair it with input metrics, guardrails, and cuts across important populations.
An event should name a business fact
button_clicked is tied to UI, while report_exported describes durable intent. Business semantics survive redesigns better than selectors.
The denominator defines the metric
Ten purchases among 100 exposed users and ten among 20 visitors are different conversion rates. Keep eligible non-converters with LEFT JOIN.
Retention needs a meaningful return event
Opening the app may be accidental. For a publishing editor, another save or publication may better represent value.
Calendar and rolling retention differ
Day-7 calendar asks about a specific period; rolling asks whether the user returned on day seven or later. Never compare them without definitions.
Assignment is not exposure
A user can receive a stored variant without opening the screen. Triggered analysis focuses on affected units, but its trigger must be defined before results.
The influence model determines randomization
If colleagues share a workspace document, per-user randomization contaminates groups. Assign the whole workspace together.
A CI is more useful than a winner label
An interval may allow both meaningful benefit and harm. Not significant means insufficient evidence, not a proven zero effect.
Peeking changes false-positive frequency
A fixed-horizon test assumes one planned analysis. Continuous monitoring needs a sequential method or a strict stopping rule.
Multiple testing needs control
Testing 20 equal metrics and many segments makes a chance winner more likely. Declare primary metric and planned slices in advance.
Novelty can be temporary
Users may explore a new interface or initially resist it. Cover a full business cycle and keep monitoring after rollout.
Practical and statistical significance differ
With huge samples a microscopic effect may be statistically significant while failing to pay for engineering, latency, or support.
First understand which parts of Node participate in execution.
Then remove instrumentation and focus on the central mechanism.
Finally match the model to the code that produces the live trace.
A minimal model without instrumentation
type ProductEventName =
| 'project_created'
| 'project_published';
@Injectable()
export class ProductMetricsService {
constructor(private readonly db: Database) {}
capture(input: {
eventId: string;
actorId: string;
name: ProductEventName;
}) {
return this.db.query(
'INSERT INTO product_events ' +
'(event_id, user_id, event_name) ' +
'VALUES ($1, $2, $3) ' +
'ON CONFLICT (event_id) DO NOTHING',
[input.eventId, input.actorId, input.name],
);
}
}The complete code executed by the scenario
This is not an alternative example: these are the functions and files used by the Run button.
The source is generated from the real server function. Scenarios using a child process or Worker include every participating file.
function percentage(value) {
return Number((value * 100).toFixed(2));
}
export function funnel(events, orderedSteps) {
const progressByUser = new Map();
for (const event of events) {
const progress = progressByUser.get(event.userId) ?? 0;
if (event.name === orderedSteps[progress]) {
progressByUser.set(event.userId, progress + 1);
}
}
return orderedSteps.map((step, index) => {
const users = [...progressByUser.values()].filter(
(progress) => progress > index,
).length;
return { step, users };
});
}
export function retention(users, day) {
const eligible = users.filter((user) => user.observedDays >= day);
const returned = eligible.filter((user) => user.activeDays.includes(day));
return {
eligible: eligible.length,
returned: returned.length,
rate: eligible.length ? returned.length / eligible.length : 0,
};
}
export function compareBinaryExperiment(control, variant) {
const controlRate = control.conversions / control.users;
const variantRate = variant.conversions / variant.users;
const absoluteLift = variantRate - controlRate;
const standardError = Math.sqrt(
(controlRate * (1 - controlRate)) / control.users +
(variantRate * (1 - variantRate)) / variant.users,
);
const margin95 = 1.96 * standardError;
const total = control.users + variant.users;
return {
controlRate,
variantRate,
absoluteLift,
confidenceInterval95: [absoluteLift - margin95, absoluteLift + margin95],
allocation: {
control: control.users / total,
variant: variant.users / total,
},
};
}
export async function productMetricsExperiment(emit) {
const events = [
{ userId: 'u1', name: 'signup_completed' },
{ userId: 'u1', name: 'project_created' },
{ userId: 'u1', name: 'project_published' },
{ userId: 'u2', name: 'signup_completed' },
{ userId: 'u2', name: 'project_created' },
{ userId: 'u3', name: 'signup_completed' },
];
const users = [
{ observedDays: 14, activeDays: [0, 1, 7], variant: 'control' },
{ observedDays: 14, activeDays: [0, 3], variant: 'control' },
{ observedDays: 14, activeDays: [0, 7], variant: 'variant' },
{ observedDays: 3, activeDays: [0, 1], variant: 'variant' },
];
emit(
'instrumentation',
'contract',
'События используют стабильные имена, userId и server timestamp',
);
const funnelResult = funnel(events, [
'signup_completed',
'project_created',
'project_published',
]);
emit(
'funnel',
'result',
funnelResult.map(({ step, users }) => `${step}=${users}`).join(' → '),
);
const day7 = retention(users, 7);
emit(
'cohort',
'result',
`D7 retention=${percentage(day7.rate)}%; denominator=${day7.eligible}`,
);
const experiment = compareBinaryExperiment(
{ users: 5_000, conversions: 1_000 },
{ users: 5_100, conversions: 1_096 },
);
const [low, high] = experiment.confidenceInterval95.map(percentage);
emit(
'experiment',
'result',
`Control=${percentage(experiment.controlRate)}%; variant=${percentage(experiment.variantRate)}%; lift CI95=[${low}, ${high}] п.п.`,
);
emit(
'decision',
'guardrail',
'Решение требует заранее выбранной primary metric, guardrails и проверки SRM',
);
return { funnel: funnelResult, retention: day7, experiment };
}
The application instruments its own live trace: rows and timestamps are recorded by real emit(...) calls, while the scenario supplies source and lane labels. This is not a V8/libuv profiler or a direct view of their internal queues. await and Promise keep the HTTP stream open until the scenario completes.
Practical patterns worth keeping nearby
Compare the goal, code, and caveats instead of memorizing syntax without a model.
Nest: a server event contract
Prevent a client from impersonating the actor or submitting an unknown event name.
const allowedNames = [
'report_created',
'report_published',
] as const;
type ProductEventName = typeof allowedNames[number];
@Injectable()
export class ProductMetricsService {
capture(
actorId: string,
name: ProductEventName,
properties: Record<string, string | number>,
) {
return this.events.insert({
eventId: randomUUID(),
actorId,
name,
properties,
schemaVersion: 1,
});
}
}- as const preserves string literals instead of widening to string.
- ProductEventName becomes a union of allowed names.
- Actor arrives separately from authenticated context.
- A compile-time type does not replace runtime validation of external JSON.
SQL: signup funnel
Count users reaching signup → project → publish within seven days.
WITH first_steps AS (
SELECT user_id,
min(occurred_at) FILTER (
WHERE event_name = 'signup_completed'
) AS signed_up_at,
min(occurred_at) FILTER (
WHERE event_name = 'project_created'
) AS project_at,
min(occurred_at) FILTER (
WHERE event_name = 'project_published'
) AS published_at
FROM product_events
GROUP BY user_id
)
SELECT
count(*) FILTER (WHERE signed_up_at IS NOT NULL) AS signup,
count(*) FILTER (WHERE project_at >= signed_up_at) AS project,
count(*) FILTER (
WHERE published_at >= project_at
AND published_at < signed_up_at + interval '7 days'
) AS published
FROM first_steps;- WITH creates the intermediate first_steps relation.
- min finds each user's first event of a type.
- FILTER limits rows entering one aggregate.
- Conditions preserve funnel order and its time window.
SQL: weekly retention
Measure return to publishing during the week after activation.
WITH activated AS (
SELECT user_id,
date_trunc('week', min(occurred_at)) AS cohort_week
FROM product_events
WHERE event_name = 'project_published'
GROUP BY user_id
), returned AS (
SELECT DISTINCT a.user_id, a.cohort_week
FROM activated a
JOIN product_events e ON e.user_id = a.user_id
AND e.event_name = 'project_published'
AND e.occurred_at >= a.cohort_week + interval '1 week'
AND e.occurred_at < a.cohort_week + interval '2 weeks'
)
SELECT a.cohort_week,
count(*) AS activated_users,
count(r.user_id) AS retained_users,
count(r.user_id)::numeric / count(*) AS week_1_retention
FROM activated a
LEFT JOIN returned r USING (user_id, cohort_week)
GROUP BY a.cohort_week
ORDER BY a.cohort_week;- A user joins a cohort at their first publication.
- DISTINCT prevents frequent publishers from inflating the numerator.
- LEFT JOIN keeps non-retained users in the denominator.
- This definition is calendar week-1, not rolling retention.
SQL: subscription churn
Avoid dividing lost subscriptions by the entire historical customer base.
SELECT
date_trunc('month', cancelled_at) AS month,
count(*) AS cancelled,
count(*)::numeric /
nullif(max(active_at_month_start), 0) AS user_churn
FROM subscription_cancellations
GROUP BY date_trunc('month', cancelled_at)
ORDER BY month;- The denominator is active subscriptions at the start of that period.
- nullif prevents division by zero.
- Revenue churn sums lost recurring revenue instead of users.
Stable variant assignment
Give one user one variant without a central random generator.
function assignVariant(
experimentKey: string,
userId: string,
) {
const bucket = createHash('sha256')
.update(experimentKey + ':' + userId)
.digest()
.readUInt32BE(0) % 10_000;
return bucket < 5_000 ? 'control' : 'treatment';
}- Experiment key separates independent tests.
- A deterministic hash repeats assignment on another Nest instance.
- A production implementation versions allocation and stores exposure.
- Use accountId for an account-level feature.
SQL: check SRM before reading outcomes
Compare observed variant sizes with the planned allocation.
SELECT variant, count(DISTINCT user_id) AS participants
FROM experiment_exposures
WHERE experiment_key = $1
GROUP BY variant;
-- A 50/50 plan expects similar counts.
-- The experiment service runs a statistical SRM check
-- before evaluating the primary metric.- DISTINCT matches the user randomization unit.
- A material mismatch needs investigation, not manual result correction.
- The threshold depends on sample and allocation; visual inspection alone is insufficient.
Write the experiment card before launch
Make decision criteria auditable before a result is visible.
const checkoutExperiment = {
hypothesis:
'Shorter form increases completed purchases',
unit: 'user',
population: 'authenticated users entering checkout',
primaryMetric: 'purchase within 24h of exposure',
guardrails: ['payment_error_rate', 'refund_rate'],
allocation: { control: 0.5, treatment: 0.5 },
mde: 0.02,
stoppingRule: 'planned sample and full business weeks',
};- Clarify whether MDE 0.02 means absolute percentage points or relative uplift.
- Guardrails need acceptable limits, not names alone.
- A frozen plan reduces freedom to select a convenient post-hoc result.
How a learning mistake becomes an incident
A realistic service: the original code, observable failure, corrected implementation, and why the correction works.
Activation rises because an event is recorded before business success
The team calls a user activated after their first project is created. The controller sends an unawaited event before the project write finishes, and every HTTP retry creates another event.
A database failure still leaves false activation behind, while a retry doubles the events. The dashboard reports growth absent from business data and can drive the wrong product decision.
@Post()
async create(
@CurrentUser() user: User,
@Body() dto: CreateProjectDto,
) {
this.metrics.capture({
eventId: randomUUID(),
name: 'project_created',
userId: user.id,
}); // Promise is not awaited
return this.projects.create(user.id, dto);
}The event is not coupled to the project commit, and a new randomUUID on every retry cannot identify a duplicate. The method may fail after analytics has already accepted a success.
@Injectable()
export class ProjectsService {
constructor(private readonly db: Database) {}
create(userId: string, dto: CreateProjectDto) {
return this.db.transaction(async (tx) => {
const project = await tx.one(
'INSERT INTO projects (user_id, name) ' +
'VALUES ($1, $2) RETURNING id, name',
[userId, dto.name],
);
await tx.query(
'INSERT INTO product_event_outbox ' +
'(dedupe_key, event_name, user_id, properties) ' +
'VALUES ($1, $2, $3, $4::jsonb) ' +
'ON CONFLICT (dedupe_key) DO NOTHING',
[
'project_created:' + project.id,
'project_created',
userId,
JSON.stringify({ projectId: project.id }),
],
);
return project;
});
}
}The project and outbox event commit in one database transaction. A stable dedupe_key makes retries safe, and a publisher delivers only committed events to analytics.
What the unfamiliar calls from both code samples actually do.
@Injectable()- A Nest decorator that marks the class as a provider the IoC container can construct and inject into a controller.
db.transaction(callback)- Opens a transaction: either both the project and outbox event persist, or both changes roll back.
tx.one(sql, values)- An illustrative repository helper that runs parameterized SQL and requires exactly one returned row.
$1, $2 and values- PostgreSQL placeholders. Values travel separately from SQL text and positions start at one.
RETURNING- Returns fields from the row just written without a second SELECT; the event needs the new project id.
ON CONFLICT ... DO NOTHING- A retry with the same unique dedupe_key does not create a second event row.
outbox- A table of event-delivery intents. A separate worker reads committed rows and reliably forwards them to analytics.
An A/B test “wins” because every request gets a new random variant
A new checkout form is selected with Math.random() on every page load. Analysis groups only completed orders by the last variant the user saw.
One user enters both groups, exposure is duplicated, and non-buyers are absent from the denominator. The observed difference has no valid causal interpretation.
@Get('checkout')
async checkout(@CurrentUser() user: User) {
const variant = Math.random() < 0.5 ? 'A' : 'B';
await this.metrics.capture({
name: 'checkout_exposed',
userId: user.id,
properties: { variant },
});
return this.checkoutPage.render({ variant });
}
// Analysis bug: buyers only
SELECT variant, count(*)
FROM orders
GROUP BY variant;Assignment is unstable and violates the randomization unit. The query counts outcomes without all exposed participants, so it does not compute a conversion rate.
@Injectable()
export class CheckoutExperiment {
constructor(private readonly db: Database) {}
async getVariant(userId: string) {
const bucket = createHash('sha256')
.update('checkout-v2:' + userId)
.digest()
.readUInt32BE(0) % 10_000;
const variant = bucket < 5_000
? 'control'
: 'treatment';
await this.db.query(
'INSERT INTO experiment_exposures ' +
'(experiment_key, user_id, variant, exposed_at) ' +
'VALUES ($1, $2, $3, now()) ' +
'ON CONFLICT (experiment_key, user_id) DO NOTHING',
['checkout-v2', userId, variant],
);
return variant;
}
}
WITH exposed AS (
SELECT user_id, variant, min(exposed_at) AS exposed_at
FROM experiment_exposures
WHERE experiment_key = 'checkout-v2'
GROUP BY user_id, variant
), converted AS (
SELECT DISTINCT e.user_id
FROM exposed e
JOIN orders o ON o.user_id = e.user_id
AND o.created_at >= e.exposed_at
)
SELECT e.variant,
count(*) AS participants,
count(c.user_id) AS conversions,
count(c.user_id)::numeric / count(*) AS rate
FROM exposed e
LEFT JOIN converted c USING (user_id)
GROUP BY e.variant;The hash keeps each user in one group. A unique exposure provides an honest denominator, LEFT JOIN retains non-buyers, and only outcomes after first exposure count.
What the unfamiliar calls from both code samples actually do.
createHash(...).update(...)- Deterministically turns experiment key plus userId into bytes, so the same user receives the same bucket.
readUInt32BE(0) % 10_000- Reads a number from the first four hash bytes and maps it to bucket 0–9999 for percentage rollout.
UNIQUE (experiment_key, user_id)- A database constraint prevents two assignments for the same user in one experiment.
WITH exposed AS (...)- A CTE names an intermediate relation and makes the analysis denominator explicit.
LEFT JOIN- Retains every exposed user even with no matching order; a missing order is a non-conversion.
count(c.user_id)::numeric / count(*)- Divides converters by all group participants. ::numeric prevents integer division.
Common misconceptions
The myth is on the left; the accurate model is on the right.
More events always mean more understanding.
Without an owner, schema, and product question, events create contradictory definitions and storage cost.
DAU always works as a North Star.
DAU measures presence, not necessarily received value; the metric must represent this product's core outcome.
Retention is any repeat visit.
It requires an explicit cohort, return action, and time window.
If p < 0.05, the feature is definitely useful.
Inspect design, SRM, data quality, CI, effect size, guardrails, and implementation cost.
No significance means the variants are equal.
The test may lack power; the CI shows which effects remain compatible with the data.
Math.random() on every request creates an A/B test.
A unit needs stable assignment, and exposure plus outcome must link back to it correctly.
Stop as soon as the dashboard turns green.
Unplanned peeking and stopping increase false-positive decisions.
A segment discovered after the test already proves an effect.
A post-hoc segment is a new hypothesis that needs confirmation on independent data.
Explain it in your own words
If you can explain the answer without quoting documentation, your mental model is starting to take shape.
- Why does a North Star need guardrail metrics?
- How does an event differ from its properties?
- What belongs in the denominator of a conversion rate?
- How do cohort and return event define retention?
- How do user churn and revenue churn differ?
- Why is assignment not always exposure?
- How do you choose user versus account as a randomization unit?
- How do MDE, sample size, and power relate?
- What does a confidence interval add beyond a significant label?
- Why do peeking and multiple testing increase chance wins?
- What does SRM signal, and why should it be checked before outcomes?
- How does a transactional outbox protect product events?
- Why does a successful A/B test not remove post-rollout monitoring?