NNODE LOOP LABruntime observatoryNNEON · Articles in 90 languages
CONNECTING
v24.18.0linux/x64
Desired state → controllers → rollout
20

Kubernetes: Pods, Services, and reconciliation

Follow a container image through Deployment reconciliation, scheduling, readiness, Service routing, and a rolling update.

PROCESS IDcurrent server
UPTIMEsince startup
LOOP DELAY P95perf_hooks
UTILIZATIONevent loop
HTTP ROUNDTRIPbrowser → server
LIVE TRACE

Timeline

READY
0 ms
Events will appear hereRun the selected scenario
#TIMESOURCEEVENT
Waiting for an experiment…
CHAPTER 20
DEEP DIVE · FROM BASICS TO CODE

Understanding: Kubernetes: Pods, Services, and reconciliation

You can read this chapter before running the experiment. Then return to the live trace and match each concept to a real event.

FIRST, IN PLAIN LANGUAGE

Docker can start a container on one machine. Kubernetes resembles a dispatcher for a fleet: you declare that three application copies should run, and it keeps comparing reality with that goal, places Pods, replaces failures, and connects ready instances to a stable address.

TECHNICAL FOUNDATION

Kubernetes is a declarative orchestrator for container workloads. The API server stores desired objects, controllers reconcile them, the scheduler chooses a node, kubelet maintains Pod lifecycle, and a Service exposes a stable virtual endpoint for a dynamic set of Ready Pods.

Why it mattersKubernetes is not needed merely to start one container. It manages many instances and nodes through self-healing, rolling updates, service discovery, configuration, scheduling, and resource governance. The cost is another distributed system that should have a measured justification.
WHERE THE WORK RUNS
01MANIFESTdesired state
02CONTROL PLANEAPI · controllers · scheduler
03NODEkubelet · container runtime
04TRAFFICService · ready Pods
01 · GLOSSARY

Terms used in this experiment

Understand the words first, then the execution order.

01

Cluster

A control plane and worker nodes that jointly run and manage Kubernetes objects.

02

Pod

The smallest deployable Kubernetes unit: one or more closely related containers sharing networking and volumes.

03

Deployment

A stateless-Pod controller that manages replicas, ReplicaSets, and rolling updates.

04

Service

A stable network endpoint and DNS name for a dynamic Pod set selected by labels.

05

Label / selector

A label marks an object with key/value metadata; a selector connects controllers or Services to matching objects.

06

Reconciliation loop

A controller repeatedly compares desired state with observed state and acts to remove the difference.

07

Readiness probe

A traffic-readiness check; failure removes a Pod from Service endpoints without necessarily restarting it.

08

Liveness probe

A progress check whose sustained failure causes the container to restart.

09

Resource request / limit

A request informs scheduling and reservation; a limit bounds permitted runtime consumption.

10

Rolling update

A gradual replacement of old Pods with new Pods controlled by maxSurge and maxUnavailable.

02 · MECHANICS

What happens step by step

Each step maps to an observable runtime state.

  1. 01
    Submit a manifest

    kubectl sends YAML to the API server, where schema validation checks apiVersion, kind, metadata, and spec.

  2. 02
    Store desired state

    The control plane persists the Deployment and increments generation when its Pod template changes.

  3. 03
    Create a ReplicaSet

    The Deployment controller observes a difference and creates a ReplicaSet for the current revision.

  4. 04
    Schedule Pods

    The scheduler chooses nodes that satisfy requests, affinity, taints, and other placement constraints.

  5. 05
    Start containers

    Kubelet asks the container runtime to pull the image and maintains declared Pod lifecycle.

  6. 06
    Check readiness

    A readiness probe admits a Pod to Service endpoints only after the application can receive traffic.

  7. 07
    Route traffic

    A Service selector finds Ready Pods and gives clients a stable name independent of changing Pod IPs.

  8. 08
    Reconcile an update

    For a new image, the controller creates new Pods, waits for readiness, and removes old ones within strategy bounds.

03 · CONTEXT

Where the result needs context

These details explain why similar code can sometimes produce a different trace.

01

Kubernetes does not build images

CI builds and pushes an image to a registry; Kubernetes receives a reference and runs that content on nodes.

02

A Pod is not a small VM

Containers in one Pod share a network namespace, communicate through localhost, and have one Pod IP.

03

A replica is not a backup

Three stateless Pods improve process availability but do not replace data backup or a multi-zone database.

04

Readiness and liveness answer different questions

NotReady removes traffic; liveness failure restarts. Database-coupled liveness can restart an entire fleet during an outage.

05

Requests matter to the scheduler

Without requests the scheduler cannot know demand. CPU limits usually throttle; memory-limit excess can cause an OOM kill.

06

A Secret is not encrypted by its name

The object separates data from Pod specs, but base64 is not encryption; RBAC and encryption at rest are still needed.

07

A Service does not always expose the internet

ClusterIP is internal; external HTTP commonly uses an Ingress, Gateway, or LoadBalancer Service.

01
Theory

First understand which parts of Node participate in execution.

02
Simplified code

Then remove instrumentation and focus on the central mechanism.

03
Runtime code

Finally match the model to the code that produces the live trace.

04 · Simplified code

A minimal model without instrumentation

src/demos.js · educational snippetJavaScript
apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-loop-lab
spec:
  replicas: 3
  selector:
    matchLabels:
      app: node-loop-lab
  template:
    metadata:
      labels:
        app: node-loop-lab
    spec:
      containers:
        - name: app
          image: ghcr.io/example/node-loop-lab:1.0.0
          readinessProbe:
            httpGet:
              path: /api/health
              port: 3000
05 · Runtime code

The complete code executed by the scenario

This is not an alternative example: these are the functions and files used by the Run button.

ACTUAL SOURCE

The source is generated from the real server function. Scenarios using a child process or Worker include every participating file.

src/infrastructure-lab.js
infrastructure266 lines
const pause = (milliseconds) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

export const dockerfileExample = `# syntax=docker/dockerfile:1
FROM node:24-alpine AS dependencies
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci

FROM dependencies AS build
COPY . .
RUN npm run build

FROM node:24-alpine AS runtime
ENV NODE_ENV=production PORT=3000
WORKDIR /app
COPY --chown=node:node --from=build /app/.next/standalone ./
COPY --chown=node:node --from=build /app/.next/static ./.next/static
USER node
EXPOSE 3000
HEALTHCHECK CMD node -e "fetch('http://127.0.0.1:3000/api/health').then(r => { if (!r.ok) process.exit(1) })"
CMD ["node", "server.js"]`;

export const composeExample = `services:
  app:
    build:
      context: .
      target: runtime
    init: true
    ports:
      - "127.0.0.1:3000:3000"
    environment:
      DATABASE_URL: postgresql://app:password@postgres:5432/app
    depends_on:
      postgres:
        condition: service_healthy
    mem_limit: 2g
    pids_limit: 128
    restart: unless-stopped

  postgres:
    image: postgres:18-alpine
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app"]
      interval: 5s
      timeout: 3s
      retries: 12`;

export const kubernetesManifestExample = `apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-loop-lab
spec:
  replicas: 3
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: node-loop-lab
  template:
    metadata:
      labels:
        app: node-loop-lab
    spec:
      containers:
        - name: app
          image: ghcr.io/example/node-loop-lab:1.0.0
          ports:
            - name: http
              containerPort: 3000
          readinessProbe:
            httpGet:
              path: /api/health
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /api/health
              port: http
            periodSeconds: 10
            failureThreshold: 3
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: node-loop-lab
spec:
  selector:
    app: node-loop-lab
  ports:
    - name: http
      port: 80
      targetPort: http`;

function dockerStages(source) {
  const stages = [];
  for (const line of source.split('\n')) {
    const match = line.match(/^FROM\s+(\S+)(?:\s+AS\s+(\S+))?/i);
    if (match) {
      stages.push({
        image: match[1],
        name: match[2] ?? `stage-${stages.length + 1}`,
      });
    }
  }
  return stages;
}

export async function dockerBuildAndRun(emit) {
  const stages = dockerStages(dockerfileExample);

  emit(
    'build-context',
    'context',
    'Docker client собирает build context; .dockerignore исключает node_modules, .git и секреты',
  );
  await pause(15);

  emit(
    'dockerfile',
    'parse',
    `Dockerfile описывает ${stages.length} стадии: ${stages.map((stage) => stage.name).join(' → ')}`,
  );

  for (const stage of stages) {
    await pause(15);
    emit(
      'buildkit',
      'stage',
      `BuildKit строит stage ${stage.name} из immutable base image ${stage.image}`,
    );
  }

  emit(
    'cache',
    'layer',
    'COPY package*.json расположен до COPY исходников: изменение кода не инвалидирует слой npm ci',
  );
  await pause(15);
  emit(
    'image',
    'artifact',
    'Runtime image получает standalone build, но не исходный build toolchain',
  );
  await pause(15);
  emit(
    'container',
    'process',
    'Container запускает node server.js как USER node; init передаёт сигналы и убирает zombie processes',
  );
  await pause(15);
  emit(
    'network',
    'publish',
    'Port mapping 127.0.0.1:3000:3000 публикует container port только на loopback хоста',
  );
  await pause(15);
  emit(
    'health',
    'probe',
    'Healthcheck проверяет /api/health; healthy не означает, что все внешние зависимости доступны',
  );
  emit(
    'result',
    'summary',
    'Image — неизменяемый шаблон; container — запущенный process с writable layer и runtime configuration',
  );
}

function readyPods(pods) {
  return pods.filter((pod) => pod.ready);
}

export async function kubernetesReconciliation(emit) {
  const desiredReplicas = 3;
  let generation = 1;
  let pods = [
    { name: 'node-loop-lab-old-1', version: '1.0.0', ready: true },
  ];

  emit(
    'api-server',
    'desired-state',
    `Deployment принят: desired replicas=${desiredReplicas}, image=1.0.0`,
  );
  await pause(15);

  while (pods.length < desiredReplicas) {
    const pod = {
      name: `node-loop-lab-old-${pods.length + 1}`,
      version: '1.0.0',
      ready: false,
    };
    pods.push(pod);
    emit(
      'deployment-controller',
      'reconcile',
      `Actual=${pods.length - 1}, desired=${desiredReplicas}: ReplicaSet создаёт ${pod.name}`,
    );
    await pause(15);
    pod.ready = true;
    emit(
      'kubelet',
      'readiness',
      `${pod.name} прошёл readinessProbe и добавлен в endpoints Service`,
    );
  }

  emit(
    'service',
    'routing',
    `Service выбирает по label ${readyPods(pods).length} ready Pods из ${pods.length}`,
  );
  await pause(15);

  pods[1].ready = false;
  emit(
    'readiness',
    'traffic',
    `${pods[1].name} стал NotReady: container продолжает работать, но Service исключил его из трафика`,
  );
  await pause(15);
  pods[1].ready = true;

  generation += 1;
  const newPod = {
    name: `node-loop-lab-new-${generation}`,
    version: '1.1.0',
    ready: false,
  };
  pods.push(newPod);
  emit(
    'rolling-update',
    'surge',
    `maxSurge=1: создан ${newPod.name}, старые ready Pods пока обслуживают трафик`,
  );
  await pause(15);
  newPod.ready = true;
  pods = pods.filter((pod) => pod.name !== 'node-loop-lab-old-1');
  emit(
    'rolling-update',
    'replace',
    'Новый Pod стал Ready; controller удалил один старый Pod без снижения ready replicas',
  );

  emit(
    'scheduler',
    'resources',
    'Scheduler размещает Pod по requests; limits ограничивают runtime, но не резервируют дополнительный ресурс',
  );
  emit(
    'result',
    'summary',
    'Kubernetes непрерывно сравнивает desired и actual state; controller исправляет расхождение, а Service маршрутизирует только Ready Pods',
  );
}

The emit(...) calls become live-trace rows. await and Promise keep the HTTP stream open until the scenario completes.

06 · RECIPES

Practical patterns worth keeping nearby

Compare the goal, code, and caveats instead of memorizing syntax without a model.

01

Apply and inspect a rollout

Create objects and observe actual Deployment state.

kubectl apply -f node-loop-lab.yml
kubectl rollout status deployment/node-loop-lab
kubectl get pods -l app=node-loop-lab
kubectl describe deployment node-loop-lab
  • apply creates or declaratively updates objects.
  • rollout status waits for the new revision to become available.
  • describe exposes conditions and recent events.
02

Deployment skeleton

Maintain three interchangeable Pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-loop-lab
spec:
  replicas: 3
  selector:
    matchLabels:
      app: node-loop-lab
  template:
    metadata:
      labels:
        app: node-loop-lab
    spec:
      containers:
        - name: app
          image: ghcr.io/example/node-loop-lab:1.0.0
  • A Pod-template change creates a new Deployment revision.
  • The image reference must point to a published registry artifact.
03

Service selector

Give Pods stable DNS and a virtual IP.

apiVersion: v1
kind: Service
metadata:
  name: node-loop-lab
spec:
  selector:
    app: node-loop-lab
  ports:
    - port: 80
      targetPort: http
  • The selector must match the Pod label.
  • The Service does not depend on changing Pod IPs.
04

Three different probes

Separate slow startup, traffic readiness, and deadlock recovery.

startupProbe:
  httpGet: { path: /api/health, port: http }
  failureThreshold: 30
  periodSeconds: 2
readinessProbe:
  httpGet: { path: /api/health, port: http }
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /api/health, port: http }
  periodSeconds: 10
  failureThreshold: 3
  • Startup delays liveness and readiness until startup succeeds.
  • Readiness failure stops new traffic.
  • Liveness failure after the threshold causes a restart.
05

Requests and limits

Give the scheduler a demand signal and set a runtime boundary.

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: "1"
    memory: 1Gi
  • 250m is a quarter of one CPU core as a request.
  • Mi and Gi are binary memory units.
  • Measure values rather than copying them blindly.
06

Update and rollback

Move to an immutable version and return after an incident.

kubectl set image deployment/node-loop-lab \
  app=ghcr.io/example/node-loop-lab:1.1.0

kubectl rollout status deployment/node-loop-lab
kubectl rollout undo deployment/node-loop-lab
  • set image changes the desired Pod template.
  • A Deployment rollback does not roll back a database migration.
07

Diagnose a Pod

Distinguish application logs, object status, and node events.

kubectl get pod <pod> -o wide
kubectl logs <pod> -c app --previous
kubectl describe pod <pod>
kubectl get events --sort-by=.metadata.creationTimestamp
  • --previous reads logs from the prior container after a restart.
  • describe reveals probe, scheduling, and image-pull events.
07 · PRODUCTION CASES

How a learning mistake becomes an incident

A realistic service: the original code, observable failure, corrected implementation, and why the correction works.

CASE 01

One shared health endpoint turns a database outage into a restart storm

A Deployment uses image:latest, omits requests, and points liveness at an endpoint that returns 500 during a brief PostgreSQL outage.

INCIDENT CONTEXT

Kubelet restarts every Pod because of an external dependency. Remaining replicas receive more load, while a scheduler without requests can pack the application onto an overloaded node.

BEFOREPROBLEMATIC IMPLEMENTATION
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: api
          image: ghcr.io/example/api:latest
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            periodSeconds: 2

Liveness answers “is the database available now?” rather than “is this process irrecoverably stuck?” A moving tag hides the revision, and missing requests or strategy makes placement and rollout less predictable.

AFTERCORRECTED IMPLEMENTATION
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: ghcr.io/example/api:1.7.3
          startupProbe:
            httpGet: { path: /health/live, port: 3000 }
            failureThreshold: 30
            periodSeconds: 2
          livenessProbe:
            httpGet: { path: /health/live, port: 3000 }
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet: { path: /health/ready, port: 3000 }
            periodSeconds: 5
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits: { cpu: "1", memory: 1Gi }

The live endpoint tests process progress without requiring every dependency. The ready endpoint removes a temporarily incapable Pod from traffic. A version, rollout strategy, and resources make rollout and placement observable.

FUNCTIONS AND CONSTRUCTS

What the unfamiliar calls from both code samples actually do.

replicas: 3
Declares the desired Pod count; controllers asynchronously create or remove instances to reach that state.
matchLabels
A selector connects a Deployment to managed Pods, and the same labels let a Service discover traffic backends.
startupProbe
Provides a separate window for slow startup and delays liveness and readiness until the first success.
livenessProbe
After consecutive failures, tells kubelet to restart a container to recover application progress.
readinessProbe
Controls whether a Pod accepts new traffic; failure removes its endpoint without requiring a restart.
resources.requests / limits
Requests participate in scheduling and capacity accounting while limits set upper CPU and memory runtime bounds.
maxUnavailable / maxSurge
Bound unavailable and extra Pods while a Deployment revision is gradually replaced.
WHY THE CORRECTION WORKS

Design the probe contract as application behavior. Load-test startup, p99, and memory before production; use disruption and topology controls for planned or node failures; alert on restarts and unavailable replicas.

WHAT PRODUCTION SHOWED
  • A short database outage coincides with a spike in container restarts.
  • Pods show OOMKilled or CPU throttling without a capacity baseline.
  • Operators cannot identify the content that ran under latest.
08 · DO NOT CONFUSE

Common misconceptions

The myth is on the left; the accurate model is on the right.

MYTH

Kubernetes replaces the Dockerfile and registry.

ACTUALLY

The orchestrator runs ready images; build and supply chain remain separate CI concerns.

MYTH

A Running container means the app is ready.

ACTUALLY

The process may still be loading or unable to serve; readiness distinguishes that state.

MYTH

Liveness should verify every dependency.

ACTUALLY

An external outage is not always fixed by restart and can trigger a cascading failure.

MYTH

Using image: latest is harmless.

ACTUALLY

A moving tag makes rollout and rollback irreproducible; use an immutable version or digest.

MYTH

Three replicas guarantee high availability.

ACTUALLY

Without topology constraints they can share one node or failure zone.

MYTH

kubectl apply updates all Pods immediately.

ACTUALLY

The API changes desired state and controllers perform the rollout asynchronously.

09 · SELF-CHECK

Explain it in your own words

If you can explain the answer without quoting documentation, your mental model is starting to take shape.

  1. How do a Pod, Deployment, and Service differ?
  2. Who compares desired replicas with actual replicas?
  3. Why must a Service selector match Pod labels?
  4. What happens after readiness failure versus liveness failure?
  5. How does a resource request differ from a limit?
  6. Why does latest prevent reproducible rollback?
  7. Why do three Pods not replace a PostgreSQL backup?
  8. What does kubectl apply actually do?