You can read this chapter before running the experiment. Then return to the live trace and match each concept to a real event.
A metric is a number measured repeatedly with labels and time. The application exposes current values, Prometheus retrieves and stores them as time series, and Grafana builds panels that reveal trends. Neither Prometheus nor Grafana repairs a leak.
The lab exposes Prometheus text format at /api/metrics: main process and memory child usage, Event Loop utilization and delay, active runs, and errors. The optional Docker Compose stack scrapes it every five seconds and provisions a Grafana datasource and dashboard.
Understand the words first, then the execution order.
A numeric measurement over time. A gauge moves both ways; a counter normally only increases.
Samples sharing one metric name and one unique label combination.
An HTTP request from Prometheus to a metrics endpoint for the current samples.
The number of unique label combinations. A userId or requestId label can explode storage cost.
A measurable service-quality indicator such as success ratio or request latency.
A rule over a time series that should usually require a sustained condition rather than one sample.
Each step maps to an observable runtime state.
process.memoryUsage and perf_hooks provide runtime signals while business code increments counters.
/api/metrics returns HELP, TYPE, and samples in Prometheus text format.
It stores values with timestamps in its time-series database at regular intervals.
rate(counter[window]), max_over_time, and comparisons turn samples into diagnostic signals.
The provisioned dashboard shows main and child memory, Event Loop delay, ELU, and run rates.
A sustained trend opens a runbook: workload, logs, safe-replica snapshot, retainer path, and fix.
These details explain why similar code can sometimes produce a different trace.
A large stable heap may be normal; a baseline that rises after equivalent load and GC cycles is suspicious.
Synchronous I/O or GC can produce delay, while process CPU includes work outside the main Event Loop. Correlate signals.
An absolute run counter grows with uptime. Use a windowed rate for current intensity; Prometheus handles resets.
mode and outcome have low cardinality. Put IDs, emails, stack traces, and request IDs in logs or traces.
A real service also needs HTTP RED metrics, pool and queue saturation, cgroup memory, restarts and OOMKill, alert routing, and retention policy.
First understand which parts of Node participate in execution.
Then remove instrumentation and focus on the central mechanism.
Finally match the model to the code that produces the live trace.
import { monitorEventLoopDelay } from 'node:perf_hooks';
const delay = monitorEventLoopDelay({ resolution: 20 });
delay.enable();
// Prometheus scrape:
// GET /api/metrics
// node_loop_lab_process_resident_memory_bytes 123456789
// node_loop_lab_event_loop_delay_p95_seconds 0.012
console.log(delay.percentile(95) / 1e6, 'ms');This is not an alternative example: these are the functions and files used by the Run button.
The source is generated from the real server function. Scenarios using a child process or Worker include every participating file.
import {
monitorEventLoopDelay,
performance,
} from 'node:perf_hooks';
const sleep = (ms) =>
new Promise((resolve) => setTimeout(resolve, ms));
function blockMainThread(durationMs) {
const startedAt = performance.now();
let iterations = 0;
// Намеренная блокировка для учебного сценария. В production так делать нельзя.
while (performance.now() - startedAt < durationMs) {
iterations += Math.sqrt((iterations % 10_000) + 1);
}
return Math.round(iterations);
}
async function observabilitySignals(emit) {
const histogram = monitorEventLoopDelay({ resolution: 10 });
histogram.enable();
const eluBefore = performance.eventLoopUtilization();
const memoryBefore = process.memoryUsage();
emit(
'metrics',
'sample',
`До нагрузки: RSS=${Math.round(memoryBefore.rss / 1024 / 1024)} MB, heapUsed=${Math.round(
memoryBefore.heapUsed / 1024 / 1024,
)} MB`,
);
emit(
'prometheus',
'info',
'Prometheus получает эти process/runtime/memory-lab ряды через GET /api/metrics',
);
// Даём monitorEventLoopDelay установить свой sampling timer до блокировки.
await sleep(35);
emit(
'event-loop',
'warning',
'Создаём короткую контролируемую блокировку, чтобы метрика delay получила сигнал',
);
blockMainThread(140);
await sleep(35);
const elu = performance.eventLoopUtilization(eluBefore);
const p95Ms = Number.isFinite(histogram.percentile(95))
? histogram.percentile(95) / 1e6
: 0;
const maxMs = Number.isFinite(histogram.max) ? histogram.max / 1e6 : 0;
histogram.disable();
emit(
'metrics',
'result',
`Event Loop: p95 delay=${p95Ms.toFixed(1)} мс, max=${maxMs.toFixed(
1,
)} мс, ELU=${(elu.utilization * 100).toFixed(1)}%`,
);
emit(
'grafana',
'result',
'Grafana не измеряет процесс сама: она строит панели по временным рядам, которые собрал Prometheus',
);
}The application instruments its own live trace: rows and timestamps are recorded by real emit(...) calls, while the scenario supplies source and lane labels. This is not a V8/libuv profiler or a direct view of their internal queues. await and Promise keep the HTTP stream open until the scenario completes.
A realistic service: the original code, observable failure, corrected implementation, and why the correction works.
A team wants detailed latency diagnostics and adds userId and requestId to an HTTP histogram. The dashboard is useful in staging but overloads the metrics backend in production.
Every unique label combination creates a new time series. Unbounded identifiers multiply cardinality, memory, storage, and query cost.
@Injectable()
export class MetricsInterceptor implements NestInterceptor {
intercept(context: ExecutionContext, next: CallHandler) {
const request = context.switchToHttp().getRequest();
const startedAt = performance.now();
return next.handle().pipe(finalize(() => {
httpDuration.observe({
method: request.method,
path: request.url,
userId: request.user.id,
requestId: request.id,
}, (performance.now() - startedAt) / 1_000);
}));
}
}User and request identifiers are effectively unbounded. Raw URLs may also contain IDs, making each request a new route label.
@Injectable()
export class MetricsInterceptor implements NestInterceptor {
intercept(context: ExecutionContext, next: CallHandler) {
const http = context.switchToHttp();
const request = http.getRequest();
const response = http.getResponse();
const controller = context.getClass().name;
const handler = context.getHandler().name;
const startedAt = performance.now();
return next.handle().pipe(finalize(() => {
const duration = (performance.now() - startedAt) / 1_000;
httpDuration.observe({
method: request.method,
controller,
handler,
statusClass: String(response.statusCode)[0] + 'xx',
}, duration);
logger.info({
requestId: request.id,
userId: request.user?.id,
duration,
}, 'request completed');
}));
}
}Metrics keep bounded dimensions for aggregation and alerts. High-cardinality context belongs in structured logs and traces connected through a correlation ID.
What the unfamiliar calls from both code samples actually do.
NestInterceptorExecutionContextnext.handle()finalize(callback)histogram.observe(labels, value)context.getClass() / getHandler()logger.info(fields, message)The myth is on the left; the accurate model is on the right.
Grafana collects application metrics.
In this design the app exposes, Prometheus collects and stores, and Grafana queries and visualizes.
An alert should fire on the first high sample.
A spike is often normal; define a threshold, window, duration, and runbook.
RSS can be added to heapUsed and external.
RSS is already a total resident view, while component boundaries partially overlap.
More context in labels is always better.
Unbounded values create new time series and can overload monitoring before the application.
A clean dashboard proves there is no leak.
It helps reveal symptoms. Repeatable load, profiles or snapshots, and a retainer path establish the cause.
If you can explain the answer without quoting documentation, your mental model is starting to take shape.