Kubernetes: Pods, Services, and reconciliation
Follow a container image through Deployment reconciliation, scheduling, readiness, Service routing, and a rolling update.
Timeline
Understanding: Kubernetes: Pods, Services, and reconciliation
You can read this chapter before running the experiment. Then return to the live trace and match each concept to a real event.
Docker can start a container on one machine. Kubernetes resembles a dispatcher for a fleet: you declare that three application copies should run, and it keeps comparing reality with that goal, places Pods, replaces failures, and connects ready instances to a stable address.
Kubernetes is a declarative orchestrator for container workloads. The API server stores desired objects, controllers reconcile them, the scheduler chooses a node, kubelet maintains Pod lifecycle, and a Service exposes a stable virtual endpoint for a dynamic set of Ready Pods.
Terms used in this experiment
Understand the words first, then the execution order.
Cluster
A control plane and worker nodes that jointly run and manage Kubernetes objects.
Pod
The smallest deployable Kubernetes unit: one or more closely related containers sharing networking and volumes.
Deployment
A stateless-Pod controller that manages replicas, ReplicaSets, and rolling updates.
Service
A stable network endpoint and DNS name for a dynamic Pod set selected by labels.
Label / selector
A label marks an object with key/value metadata; a selector connects controllers or Services to matching objects.
Reconciliation loop
A controller repeatedly compares desired state with observed state and acts to remove the difference.
Readiness probe
A traffic-readiness check; failure removes a Pod from Service endpoints without necessarily restarting it.
Liveness probe
A progress check whose sustained failure causes the container to restart.
Resource request / limit
A request informs scheduling and reservation; a limit bounds permitted runtime consumption.
Rolling update
A gradual replacement of old Pods with new Pods controlled by maxSurge and maxUnavailable.
What happens step by step
Each step maps to an observable runtime state.
- 01Submit a manifest
kubectl sends YAML to the API server, where schema validation checks apiVersion, kind, metadata, and spec.
- 02Store desired state
The control plane persists the Deployment and increments generation when its Pod template changes.
- 03Create a ReplicaSet
The Deployment controller observes a difference and creates a ReplicaSet for the current revision.
- 04Schedule Pods
The scheduler chooses nodes that satisfy requests, affinity, taints, and other placement constraints.
- 05Start containers
Kubelet asks the container runtime to pull the image and maintains declared Pod lifecycle.
- 06Check readiness
A readiness probe admits a Pod to Service endpoints only after the application can receive traffic.
- 07Route traffic
A Service selector finds Ready Pods and gives clients a stable name independent of changing Pod IPs.
- 08Reconcile an update
For a new image, the controller creates new Pods, waits for readiness, and removes old ones within strategy bounds.
Where the result needs context
These details explain why similar code can sometimes produce a different trace.
Kubernetes does not build images
CI builds and pushes an image to a registry; Kubernetes receives a reference and runs that content on nodes.
A Pod is not a small VM
Containers in one Pod share a network namespace, communicate through localhost, and have one Pod IP.
A replica is not a backup
Three stateless Pods improve process availability but do not replace data backup or a multi-zone database.
Readiness and liveness answer different questions
NotReady removes traffic; liveness failure restarts. Database-coupled liveness can restart an entire fleet during an outage.
Requests matter to the scheduler
Without requests the scheduler cannot know demand. CPU limits usually throttle; memory-limit excess can cause an OOM kill.
A Secret is not encrypted by its name
The object separates data from Pod specs, but base64 is not encryption; RBAC and encryption at rest are still needed.
A Service does not always expose the internet
ClusterIP is internal; external HTTP commonly uses an Ingress, Gateway, or LoadBalancer Service.
First understand which parts of Node participate in execution.
Then remove instrumentation and focus on the central mechanism.
Finally match the model to the code that produces the live trace.
A minimal model without instrumentation
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-loop-lab
spec:
replicas: 3
selector:
matchLabels:
app: node-loop-lab
template:
metadata:
labels:
app: node-loop-lab
spec:
containers:
- name: app
image: ghcr.io/example/node-loop-lab:1.0.0
readinessProbe:
httpGet:
path: /api/health
port: 3000The complete code executed by the scenario
This is not an alternative example: these are the functions and files used by the Run button.
The source is generated from the real server function. Scenarios using a child process or Worker include every participating file.
const pause = (milliseconds) =>
new Promise((resolve) => setTimeout(resolve, milliseconds));
export const dockerfileExample = `# syntax=docker/dockerfile:1
FROM node:24-alpine AS dependencies
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
FROM dependencies AS build
COPY . .
RUN npm run build
FROM node:24-alpine AS runtime
ENV NODE_ENV=production PORT=3000
WORKDIR /app
COPY --chown=node:node --from=build /app/.next/standalone ./
COPY --chown=node:node --from=build /app/.next/static ./.next/static
USER node
EXPOSE 3000
HEALTHCHECK CMD node -e "fetch('http://127.0.0.1:3000/api/health').then(r => { if (!r.ok) process.exit(1) })"
CMD ["node", "server.js"]`;
export const composeExample = `services:
app:
build:
context: .
target: runtime
init: true
ports:
- "127.0.0.1:3000:3000"
environment:
DATABASE_URL: postgresql://app:password@postgres:5432/app
depends_on:
postgres:
condition: service_healthy
mem_limit: 2g
pids_limit: 128
restart: unless-stopped
postgres:
image: postgres:18-alpine
healthcheck:
test: ["CMD-SHELL", "pg_isready -U app"]
interval: 5s
timeout: 3s
retries: 12`;
export const kubernetesManifestExample = `apiVersion: apps/v1
kind: Deployment
metadata:
name: node-loop-lab
spec:
replicas: 3
strategy:
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
selector:
matchLabels:
app: node-loop-lab
template:
metadata:
labels:
app: node-loop-lab
spec:
containers:
- name: app
image: ghcr.io/example/node-loop-lab:1.0.0
ports:
- name: http
containerPort: 3000
readinessProbe:
httpGet:
path: /api/health
port: http
periodSeconds: 5
livenessProbe:
httpGet:
path: /api/health
port: http
periodSeconds: 10
failureThreshold: 3
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: node-loop-lab
spec:
selector:
app: node-loop-lab
ports:
- name: http
port: 80
targetPort: http`;
function dockerStages(source) {
const stages = [];
for (const line of source.split('\n')) {
const match = line.match(/^FROM\s+(\S+)(?:\s+AS\s+(\S+))?/i);
if (match) {
stages.push({
image: match[1],
name: match[2] ?? `stage-${stages.length + 1}`,
});
}
}
return stages;
}
export async function dockerBuildAndRun(emit) {
const stages = dockerStages(dockerfileExample);
emit(
'build-context',
'context',
'Docker client собирает build context; .dockerignore исключает node_modules, .git и секреты',
);
await pause(15);
emit(
'dockerfile',
'parse',
`Dockerfile описывает ${stages.length} стадии: ${stages.map((stage) => stage.name).join(' → ')}`,
);
for (const stage of stages) {
await pause(15);
emit(
'buildkit',
'stage',
`BuildKit строит stage ${stage.name} из immutable base image ${stage.image}`,
);
}
emit(
'cache',
'layer',
'COPY package*.json расположен до COPY исходников: изменение кода не инвалидирует слой npm ci',
);
await pause(15);
emit(
'image',
'artifact',
'Runtime image получает standalone build, но не исходный build toolchain',
);
await pause(15);
emit(
'container',
'process',
'Container запускает node server.js как USER node; init передаёт сигналы и убирает zombie processes',
);
await pause(15);
emit(
'network',
'publish',
'Port mapping 127.0.0.1:3000:3000 публикует container port только на loopback хоста',
);
await pause(15);
emit(
'health',
'probe',
'Healthcheck проверяет /api/health; healthy не означает, что все внешние зависимости доступны',
);
emit(
'result',
'summary',
'Image — неизменяемый шаблон; container — запущенный process с writable layer и runtime configuration',
);
}
function readyPods(pods) {
return pods.filter((pod) => pod.ready);
}
export async function kubernetesReconciliation(emit) {
const desiredReplicas = 3;
let generation = 1;
let pods = [
{ name: 'node-loop-lab-old-1', version: '1.0.0', ready: true },
];
emit(
'api-server',
'desired-state',
`Deployment принят: desired replicas=${desiredReplicas}, image=1.0.0`,
);
await pause(15);
while (pods.length < desiredReplicas) {
const pod = {
name: `node-loop-lab-old-${pods.length + 1}`,
version: '1.0.0',
ready: false,
};
pods.push(pod);
emit(
'deployment-controller',
'reconcile',
`Actual=${pods.length - 1}, desired=${desiredReplicas}: ReplicaSet создаёт ${pod.name}`,
);
await pause(15);
pod.ready = true;
emit(
'kubelet',
'readiness',
`${pod.name} прошёл readinessProbe и добавлен в endpoints Service`,
);
}
emit(
'service',
'routing',
`Service выбирает по label ${readyPods(pods).length} ready Pods из ${pods.length}`,
);
await pause(15);
pods[1].ready = false;
emit(
'readiness',
'traffic',
`${pods[1].name} стал NotReady: container продолжает работать, но Service исключил его из трафика`,
);
await pause(15);
pods[1].ready = true;
generation += 1;
const newPod = {
name: `node-loop-lab-new-${generation}`,
version: '1.1.0',
ready: false,
};
pods.push(newPod);
emit(
'rolling-update',
'surge',
`maxSurge=1: создан ${newPod.name}, старые ready Pods пока обслуживают трафик`,
);
await pause(15);
newPod.ready = true;
pods = pods.filter((pod) => pod.name !== 'node-loop-lab-old-1');
emit(
'rolling-update',
'replace',
'Новый Pod стал Ready; controller удалил один старый Pod без снижения ready replicas',
);
emit(
'scheduler',
'resources',
'Scheduler размещает Pod по requests; limits ограничивают runtime, но не резервируют дополнительный ресурс',
);
emit(
'result',
'summary',
'Kubernetes непрерывно сравнивает desired и actual state; controller исправляет расхождение, а Service маршрутизирует только Ready Pods',
);
}
The emit(...) calls become live-trace rows. await and Promise keep the HTTP stream open until the scenario completes.
Practical patterns worth keeping nearby
Compare the goal, code, and caveats instead of memorizing syntax without a model.
Apply and inspect a rollout
Create objects and observe actual Deployment state.
kubectl apply -f node-loop-lab.yml
kubectl rollout status deployment/node-loop-lab
kubectl get pods -l app=node-loop-lab
kubectl describe deployment node-loop-lab- apply creates or declaratively updates objects.
- rollout status waits for the new revision to become available.
- describe exposes conditions and recent events.
Deployment skeleton
Maintain three interchangeable Pods.
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-loop-lab
spec:
replicas: 3
selector:
matchLabels:
app: node-loop-lab
template:
metadata:
labels:
app: node-loop-lab
spec:
containers:
- name: app
image: ghcr.io/example/node-loop-lab:1.0.0- A Pod-template change creates a new Deployment revision.
- The image reference must point to a published registry artifact.
Service selector
Give Pods stable DNS and a virtual IP.
apiVersion: v1
kind: Service
metadata:
name: node-loop-lab
spec:
selector:
app: node-loop-lab
ports:
- port: 80
targetPort: http- The selector must match the Pod label.
- The Service does not depend on changing Pod IPs.
Three different probes
Separate slow startup, traffic readiness, and deadlock recovery.
startupProbe:
httpGet: { path: /api/health, port: http }
failureThreshold: 30
periodSeconds: 2
readinessProbe:
httpGet: { path: /api/health, port: http }
periodSeconds: 5
livenessProbe:
httpGet: { path: /api/health, port: http }
periodSeconds: 10
failureThreshold: 3- Startup delays liveness and readiness until startup succeeds.
- Readiness failure stops new traffic.
- Liveness failure after the threshold causes a restart.
Requests and limits
Give the scheduler a demand signal and set a runtime boundary.
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi- 250m is a quarter of one CPU core as a request.
- Mi and Gi are binary memory units.
- Measure values rather than copying them blindly.
Update and rollback
Move to an immutable version and return after an incident.
kubectl set image deployment/node-loop-lab \
app=ghcr.io/example/node-loop-lab:1.1.0
kubectl rollout status deployment/node-loop-lab
kubectl rollout undo deployment/node-loop-lab- set image changes the desired Pod template.
- A Deployment rollback does not roll back a database migration.
Diagnose a Pod
Distinguish application logs, object status, and node events.
kubectl get pod <pod> -o wide
kubectl logs <pod> -c app --previous
kubectl describe pod <pod>
kubectl get events --sort-by=.metadata.creationTimestamp- --previous reads logs from the prior container after a restart.
- describe reveals probe, scheduling, and image-pull events.
How a learning mistake becomes an incident
A realistic service: the original code, observable failure, corrected implementation, and why the correction works.
One shared health endpoint turns a database outage into a restart storm
A Deployment uses image:latest, omits requests, and points liveness at an endpoint that returns 500 during a brief PostgreSQL outage.
Kubelet restarts every Pod because of an external dependency. Remaining replicas receive more load, while a scheduler without requests can pack the application onto an overloaded node.
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 3
template:
spec:
containers:
- name: api
image: ghcr.io/example/api:latest
livenessProbe:
httpGet:
path: /health
port: 3000
periodSeconds: 2Liveness answers “is the database available now?” rather than “is this process irrecoverably stuck?” A moving tag hides the revision, and missing requests or strategy makes placement and rollout less predictable.
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 3
strategy:
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
image: ghcr.io/example/api:1.7.3
startupProbe:
httpGet: { path: /health/live, port: 3000 }
failureThreshold: 30
periodSeconds: 2
livenessProbe:
httpGet: { path: /health/live, port: 3000 }
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet: { path: /health/ready, port: 3000 }
periodSeconds: 5
resources:
requests: { cpu: 250m, memory: 256Mi }
limits: { cpu: "1", memory: 1Gi }The live endpoint tests process progress without requiring every dependency. The ready endpoint removes a temporarily incapable Pod from traffic. A version, rollout strategy, and resources make rollout and placement observable.
What the unfamiliar calls from both code samples actually do.
replicas: 3- Declares the desired Pod count; controllers asynchronously create or remove instances to reach that state.
matchLabels- A selector connects a Deployment to managed Pods, and the same labels let a Service discover traffic backends.
startupProbe- Provides a separate window for slow startup and delays liveness and readiness until the first success.
livenessProbe- After consecutive failures, tells kubelet to restart a container to recover application progress.
readinessProbe- Controls whether a Pod accepts new traffic; failure removes its endpoint without requiring a restart.
resources.requests / limits- Requests participate in scheduling and capacity accounting while limits set upper CPU and memory runtime bounds.
maxUnavailable / maxSurge- Bound unavailable and extra Pods while a Deployment revision is gradually replaced.
Common misconceptions
The myth is on the left; the accurate model is on the right.
Kubernetes replaces the Dockerfile and registry.
The orchestrator runs ready images; build and supply chain remain separate CI concerns.
A Running container means the app is ready.
The process may still be loading or unable to serve; readiness distinguishes that state.
Liveness should verify every dependency.
An external outage is not always fixed by restart and can trigger a cascading failure.
Using image: latest is harmless.
A moving tag makes rollout and rollback irreproducible; use an immutable version or digest.
Three replicas guarantee high availability.
Without topology constraints they can share one node or failure zone.
kubectl apply updates all Pods immediately.
The API changes desired state and controllers perform the rollout asynchronously.
Explain it in your own words
If you can explain the answer without quoting documentation, your mental model is starting to take shape.
- How do a Pod, Deployment, and Service differ?
- Who compares desired replicas with actual replicas?
- Why must a Service selector match Pod labels?
- What happens after readiness failure versus liveness failure?
- How does a resource request differ from a limit?
- Why does latest prevent reproducible rollback?
- Why do three Pods not replace a PostgreSQL backup?
- What does kubectl apply actually do?