Prometheus and Grafana
Connect process memory, Event Loop delay, and the controlled leak to a real Prometheus endpoint and provisioned Grafana dashboard.
Timeline
Understanding: Prometheus and Grafana
You can read this chapter before running the experiment. Then return to the live trace and match each concept to a real event.
A metric is a number measured repeatedly with labels and time. The application exposes current values, Prometheus retrieves and stores them as time series, and Grafana builds panels that reveal trends. Neither Prometheus nor Grafana repairs a leak.
The lab exposes Prometheus text format at /api/metrics: main process and memory child usage, Event Loop utilization and delay, active runs, and errors. The optional Docker Compose stack scrapes it every five seconds and provisions a Grafana datasource and dashboard.
Terms used in this experiment
Understand the words first, then the execution order.
Metric
A numeric measurement over time. A gauge moves both ways; a counter normally only increases.
Time series
Samples sharing one metric name and one unique label combination.
Scrape
An HTTP request from Prometheus to a metrics endpoint for the current samples.
Cardinality
The number of unique label combinations. A userId or requestId label can explode storage cost.
SLI
A measurable service-quality indicator such as success ratio or request latency.
Alert
A rule over a time series that should usually require a sustained condition rather than one sample.
What happens step by step
Each step maps to an observable runtime state.
- 01The app measures
process.memoryUsage and perf_hooks provide runtime signals while business code increments counters.
- 02The endpoint exposes
/api/metrics returns HELP, TYPE, and samples in Prometheus text format.
- 03Prometheus scrapes
It stores values with timestamps in its time-series database at regular intervals.
- 04PromQL calculates
rate(counter[window]), max_over_time, and comparisons turn samples into diagnostic signals.
- 05Grafana visualizes
The provisioned dashboard shows main and child memory, Event Loop delay, ELU, and run rates.
- 06An alert starts an investigation
A sustained trend opens a runbook: workload, logs, safe-replica snapshot, retainer path, and fix.
Where the result needs context
These details explain why similar code can sometimes produce a different trace.
Heap slope matters more than one number
A large stable heap may be normal; a baseline that rises after equivalent load and GC cycles is suspicious.
Event Loop delay and CPU are not interchangeable
Synchronous I/O or GC can produce delay, while process CPU includes work outside the main Event Loop. Correlate signals.
Read counters with rate or increase
An absolute run counter grows with uptime. Use a windowed rate for current intensity; Prometheus handles resets.
Keep labels bounded
mode and outcome have low cardinality. Put IDs, emails, stack traces, and request IDs in logs or traces.
A local dashboard is not a complete production system
A real service also needs HTTP RED metrics, pool and queue saturation, cgroup memory, restarts and OOMKill, alert routing, and retention policy.
First understand which parts of Node participate in execution.
Then remove instrumentation and focus on the central mechanism.
Finally match the model to the code that produces the live trace.
A minimal model without instrumentation
import { monitorEventLoopDelay } from 'node:perf_hooks';
const delay = monitorEventLoopDelay({ resolution: 20 });
delay.enable();
// Prometheus scrape:
// GET /api/metrics
// node_loop_lab_process_resident_memory_bytes 123456789
// node_loop_lab_event_loop_delay_p95_seconds 0.012
console.log(delay.percentile(95) / 1e6, 'ms');The complete code executed by the scenario
This is not an alternative example: these are the functions and files used by the Run button.
The source is generated from the real server function. Scenarios using a child process or Worker include every participating file.
import {
monitorEventLoopDelay,
performance,
} from 'node:perf_hooks';
const sleep = (ms) =>
new Promise((resolve) => setTimeout(resolve, ms));
function blockMainThread(durationMs) {
const startedAt = performance.now();
let iterations = 0;
// Намеренная блокировка для учебного сценария. В production так делать нельзя.
while (performance.now() - startedAt < durationMs) {
iterations += Math.sqrt((iterations % 10_000) + 1);
}
return Math.round(iterations);
}
async function observabilitySignals(emit) {
const histogram = monitorEventLoopDelay({ resolution: 10 });
histogram.enable();
const eluBefore = performance.eventLoopUtilization();
const memoryBefore = process.memoryUsage();
emit(
'metrics',
'sample',
`До нагрузки: RSS=${Math.round(memoryBefore.rss / 1024 / 1024)} MB, heapUsed=${Math.round(
memoryBefore.heapUsed / 1024 / 1024,
)} MB`,
);
emit(
'prometheus',
'info',
'Prometheus получает эти process/runtime/memory-lab ряды через GET /api/metrics',
);
// Даём monitorEventLoopDelay установить свой sampling timer до блокировки.
await sleep(35);
emit(
'event-loop',
'warning',
'Создаём короткую контролируемую блокировку, чтобы метрика delay получила сигнал',
);
blockMainThread(140);
await sleep(35);
const elu = performance.eventLoopUtilization(eluBefore);
const p95Ms = Number.isFinite(histogram.percentile(95))
? histogram.percentile(95) / 1e6
: 0;
const maxMs = Number.isFinite(histogram.max) ? histogram.max / 1e6 : 0;
histogram.disable();
emit(
'metrics',
'result',
`Event Loop: p95 delay=${p95Ms.toFixed(1)} мс, max=${maxMs.toFixed(
1,
)} мс, ELU=${(elu.utilization * 100).toFixed(1)}%`,
);
emit(
'grafana',
'result',
'Grafana не измеряет процесс сама: она строит панели по временным рядам, которые собрал Prometheus',
);
}The emit(...) calls become live-trace rows. await and Promise keep the HTTP stream open until the scenario completes.
Common misconceptions
The myth is on the left; the accurate model is on the right.
Grafana collects application metrics.
In this design the app exposes, Prometheus collects and stores, and Grafana queries and visualizes.
An alert should fire on the first high sample.
A spike is often normal; define a threshold, window, duration, and runbook.
RSS can be added to heapUsed and external.
RSS is already a total resident view, while component boundaries partially overlap.
More context in labels is always better.
Unbounded values create new time series and can overload monitoring before the application.
A clean dashboard proves there is no leak.
It helps reveal symptoms. Repeatable load, profiles or snapshots, and a retainer path establish the cause.
Explain it in your own words
If you can explain the answer without quoting documentation, your mental model is starting to take shape.
- How can time series distinguish a useful bounded cache from a leak?
- Why should requestId not be a Prometheus label?
- Which runbook should open when heap and Event Loop delay grow together?
- Which HTTP metrics are still missing before this lab could support a real SLO?