NNODE LOOP LABruntime observatoryNNEON · Articles in 90 languages
CONNECTING
v24.18.0linux/x64
Metrics → time series → investigation
10

Prometheus and Grafana

Connect process memory, Event Loop delay, and the controlled leak to a real Prometheus endpoint and provisioned Grafana dashboard.

PROCESS IDcurrent server
UPTIMEsince startup
LOOP DELAY P95perf_hooks
UTILIZATIONevent loop
HTTP ROUNDTRIPbrowser → server
LIVE TRACE

Timeline

READY
0 ms
Events will appear hereRun the selected scenario
#TIMESOURCEEVENT
Waiting for an experiment…
CHAPTER 10
DEEP DIVE · FROM BASICS TO CODE

Understanding: Prometheus and Grafana

You can read this chapter before running the experiment. Then return to the live trace and match each concept to a real event.

FIRST, IN PLAIN LANGUAGE

A metric is a number measured repeatedly with labels and time. The application exposes current values, Prometheus retrieves and stores them as time series, and Grafana builds panels that reveal trends. Neither Prometheus nor Grafana repairs a leak.

TECHNICAL FOUNDATION

The lab exposes Prometheus text format at /api/metrics: main process and memory child usage, Event Loop utilization and delay, active runs, and errors. The optional Docker Compose stack scrapes it every five seconds and provisions a Grafana datasource and dashboard.

Why it mattersA heap snapshot answers “what retains memory now,” while monitoring answers “when did growth begin, under which load, and did the release fix it?” Production diagnosis needs both, plus logs and traces.
WHERE THE WORK RUNS
01YOUR JS CODEfunctions · callbacks
02NODE APIsfs · crypto · timers
03V8 + LIBUVheap · loop · pool
04OPERATING SYSTEMI/O · threads · memory
01 · GLOSSARY

Terms used in this experiment

Understand the words first, then the execution order.

01

Metric

A numeric measurement over time. A gauge moves both ways; a counter normally only increases.

02

Time series

Samples sharing one metric name and one unique label combination.

03

Scrape

An HTTP request from Prometheus to a metrics endpoint for the current samples.

04

Cardinality

The number of unique label combinations. A userId or requestId label can explode storage cost.

05

SLI

A measurable service-quality indicator such as success ratio or request latency.

06

Alert

A rule over a time series that should usually require a sustained condition rather than one sample.

02 · MECHANICS

What happens step by step

Each step maps to an observable runtime state.

  1. 01
    The app measures

    process.memoryUsage and perf_hooks provide runtime signals while business code increments counters.

  2. 02
    The endpoint exposes

    /api/metrics returns HELP, TYPE, and samples in Prometheus text format.

  3. 03
    Prometheus scrapes

    It stores values with timestamps in its time-series database at regular intervals.

  4. 04
    PromQL calculates

    rate(counter[window]), max_over_time, and comparisons turn samples into diagnostic signals.

  5. 05
    Grafana visualizes

    The provisioned dashboard shows main and child memory, Event Loop delay, ELU, and run rates.

  6. 06
    An alert starts an investigation

    A sustained trend opens a runbook: workload, logs, safe-replica snapshot, retainer path, and fix.

03 · CONTEXT

Where the result needs context

These details explain why similar code can sometimes produce a different trace.

01

Heap slope matters more than one number

A large stable heap may be normal; a baseline that rises after equivalent load and GC cycles is suspicious.

02

Event Loop delay and CPU are not interchangeable

Synchronous I/O or GC can produce delay, while process CPU includes work outside the main Event Loop. Correlate signals.

03

Read counters with rate or increase

An absolute run counter grows with uptime. Use a windowed rate for current intensity; Prometheus handles resets.

04

Keep labels bounded

mode and outcome have low cardinality. Put IDs, emails, stack traces, and request IDs in logs or traces.

05

A local dashboard is not a complete production system

A real service also needs HTTP RED metrics, pool and queue saturation, cgroup memory, restarts and OOMKill, alert routing, and retention policy.

01
Theory

First understand which parts of Node participate in execution.

02
Simplified code

Then remove instrumentation and focus on the central mechanism.

03
Runtime code

Finally match the model to the code that produces the live trace.

04 · Simplified code

A minimal model without instrumentation

src/demos.js · educational snippetJavaScript
import { monitorEventLoopDelay } from 'node:perf_hooks';

const delay = monitorEventLoopDelay({ resolution: 20 });
delay.enable();

// Prometheus scrape:
// GET /api/metrics
// node_loop_lab_process_resident_memory_bytes 123456789
// node_loop_lab_event_loop_delay_p95_seconds 0.012

console.log(delay.percentile(95) / 1e6, 'ms');
05 · Runtime code

The complete code executed by the scenario

This is not an alternative example: these are the functions and files used by the Run button.

ACTUAL SOURCE

The source is generated from the real server function. Scenarios using a child process or Worker include every participating file.

src/demos.js
scenario69 lines
import {
  monitorEventLoopDelay,
  performance,
} from 'node:perf_hooks';

const sleep = (ms) =>
  new Promise((resolve) => setTimeout(resolve, ms));

function blockMainThread(durationMs) {
  const startedAt = performance.now();
  let iterations = 0;

  // Намеренная блокировка для учебного сценария. В production так делать нельзя.
  while (performance.now() - startedAt < durationMs) {
    iterations += Math.sqrt((iterations % 10_000) + 1);
  }

  return Math.round(iterations);
}

async function observabilitySignals(emit) {
  const histogram = monitorEventLoopDelay({ resolution: 10 });
  histogram.enable();
  const eluBefore = performance.eventLoopUtilization();
  const memoryBefore = process.memoryUsage();

  emit(
    'metrics',
    'sample',
    `До нагрузки: RSS=${Math.round(memoryBefore.rss / 1024 / 1024)} MB, heapUsed=${Math.round(
      memoryBefore.heapUsed / 1024 / 1024,
    )} MB`,
  );
  emit(
    'prometheus',
    'info',
    'Prometheus получает эти process/runtime/memory-lab ряды через GET /api/metrics',
  );

  // Даём monitorEventLoopDelay установить свой sampling timer до блокировки.
  await sleep(35);
  emit(
    'event-loop',
    'warning',
    'Создаём короткую контролируемую блокировку, чтобы метрика delay получила сигнал',
  );
  blockMainThread(140);
  await sleep(35);

  const elu = performance.eventLoopUtilization(eluBefore);
  const p95Ms = Number.isFinite(histogram.percentile(95))
    ? histogram.percentile(95) / 1e6
    : 0;
  const maxMs = Number.isFinite(histogram.max) ? histogram.max / 1e6 : 0;
  histogram.disable();

  emit(
    'metrics',
    'result',
    `Event Loop: p95 delay=${p95Ms.toFixed(1)} мс, max=${maxMs.toFixed(
      1,
    )} мс, ELU=${(elu.utilization * 100).toFixed(1)}%`,
  );
  emit(
    'grafana',
    'result',
    'Grafana не измеряет процесс сама: она строит панели по временным рядам, которые собрал Prometheus',
  );
}

The emit(...) calls become live-trace rows. await and Promise keep the HTTP stream open until the scenario completes.

06 · DO NOT CONFUSE

Common misconceptions

The myth is on the left; the accurate model is on the right.

MYTH

Grafana collects application metrics.

ACTUALLY

In this design the app exposes, Prometheus collects and stores, and Grafana queries and visualizes.

MYTH

An alert should fire on the first high sample.

ACTUALLY

A spike is often normal; define a threshold, window, duration, and runbook.

MYTH

RSS can be added to heapUsed and external.

ACTUALLY

RSS is already a total resident view, while component boundaries partially overlap.

MYTH

More context in labels is always better.

ACTUALLY

Unbounded values create new time series and can overload monitoring before the application.

MYTH

A clean dashboard proves there is no leak.

ACTUALLY

It helps reveal symptoms. Repeatable load, profiles or snapshots, and a retainer path establish the cause.

07 · SELF-CHECK

Explain it in your own words

If you can explain the answer without quoting documentation, your mental model is starting to take shape.

  1. How can time series distinguish a useful bounded cache from a leak?
  2. Why should requestId not be a Prometheus label?
  3. Which runbook should open when heap and Event Loop delay grow together?
  4. Which HTTP metrics are still missing before this lab could support a real SLO?