Microservices: boundaries, messages, and failures
Understand why services are separated, how commands and events cross a network, and which failure modes distributed systems introduce.
Timeline
Understanding: Microservices: boundaries, messages, and failures
You can read this chapter before running the experiment. Then return to the live trace and match each concept to a real event.
A monolith resembles one large office: teams communicate quickly inside one building, but a move or outage affects everyone. Microservices are independent offices with explicit responsibilities. They communicate over a network, so every conversation can be delayed, duplicated, or lost.
A microservice is an independently deployable application boundary around a business capability and its owned data. Services communicate synchronously with request-response or asynchronously with messages and events. Nest provides transport abstraction and handler decorators, but network failures, contract evolution, delivery guarantees, and cross-service observability remain application concerns.
Terms used in this experiment
Understand the words first, then the execution order.
Service boundary
A deployment, responsibility, and data-ownership boundary around a business capability.
Request-response
A producer sends a request or command and expects one answer or error before a deadline.
Event
A fact that already happened, such as OrderCreated; the publisher does not require one consumer’s business response.
Message broker
Infrastructure that accepts, stores, and delivers messages, such as Kafka, RabbitMQ, NATS, or Redis Streams.
Delivery semantics
Whether delivery is at-most-once, at-least-once, or made practically exactly-once through constraints and idempotency.
Idempotency
Repeating an operation with the same key does not produce a second business effect.
Eventual consistency
Services may temporarily observe different state versions but converge after messages are processed.
Distributed trace
A connected request and message path across processes using trace and span identifiers.
What happens step by step
Each step maps to an observable runtime state.
- 01Find a capability
Draw a boundary around business ownership such as orders, inventory, or payments, not technical folders.
- 02Define a contract
Include version, operation or correlation identifiers, and a minimal payload that does not leak internal schemas.
- 03Choose interaction
Use request-response for an immediate answer and an event for independent reactions to an established fact.
- 04Cross the transport
Serialization turns a message into bytes; transport delivers it and the consumer deserializes it.
- 05Handle failure
Timeout, retry, circuit-breaker, and dead-letter policies bound partial failure.
- 06Make repeats safe
Persist an operation ID or use a UNIQUE constraint so duplicate delivery does not repeat a side effect.
- 07Connect observability
Propagate correlation and trace context through every message boundary.
Where the result needs context
These details explain why similar code can sometimes produce a different trace.
A microservice is not merely a small REST API
Code size is secondary; independent business ownership, deployment, and protected data matter.
The network is part of the program
A server can complete while the client times out, so “no response” does not mean “not executed.”
At-least-once implies duplicates
A broker may redeliver after an acknowledgement failure; consumers must be idempotent.
Published events cannot be edited in place
Independent consumers already depend on the contract, so use compatible fields, versioning, and migrations.
Database per service describes ownership
Databases may share a physical cluster, but services should not directly mutate another owner’s tables.
TCP is a teaching transport here
The lab demonstrates a real Nest boundary without infrastructure; durable production delivery may need Kafka, RabbitMQ, or NATS.
First understand which parts of Node participate in execution.
Then remove instrumentation and focus on the central mechanism.
Finally match the model to the code that produces the live trace.
A minimal model without instrumentation
@Controller()
export class InventoryMessages {
@MessagePattern({ cmd: 'inventory.reserve' })
reserve(@Payload() command: ReserveInventoryCommand) {
return this.inventory.reserve(command);
}
}
const reservation = await firstValueFrom(
inventoryClient
.send({ cmd: 'inventory.reserve' }, command)
.pipe(timeout(2_000)),
);The complete code executed by the scenario
This is not an alternative example: these are the functions and files used by the Run button.
The source is generated from the real server function. Scenarios using a child process or Worker include every participating file.
import 'reflect-metadata';
import net from 'node:net';
import {
Controller,
Dependencies,
Module,
} from '@nestjs/common';
import { NestFactory } from '@nestjs/core';
import {
ClientProxyFactory,
EventPattern,
MessagePattern,
Payload,
RpcException,
Transport,
} from '@nestjs/microservices';
import {
firstValueFrom,
lastValueFrom,
timeout,
} from 'rxjs';
const LAB_TRACE = Symbol('MICROSERVICES_LAB_TRACE');
async function reserveTcpPort() {
const server = net.createServer();
await new Promise((resolve, reject) => {
server.once('error', reject);
server.listen(0, '127.0.0.1', resolve);
});
const address = server.address();
const port = typeof address === 'object' ? address.port : null;
await new Promise((resolve) => server.close(resolve));
if (!port) throw new Error('Не удалось выбрать локальный TCP-порт');
return port;
}
class InventoryMessageController {
constructor(trace) {
this.trace = trace;
this.reservations = new Map();
}
reserve(command) {
this.trace.emit(
'consumer',
'request',
`Inventory получил command inventory.reserve для order=${command.orderId}`,
);
const existing = this.reservations.get(command.operationId);
if (existing) {
this.trace.emit(
'idempotency',
'duplicate',
`Повтор operationId=${command.operationId} вернул прежний reservation`,
);
return { ...existing, reused: true };
}
if (command.sku === 'sold-out') {
throw new RpcException({
code: 'OUT_OF_STOCK',
message: 'Товар закончился',
});
}
const reservation = {
reservationId: `reservation-${this.reservations.size + 1}`,
operationId: command.operationId,
reused: false,
};
this.reservations.set(command.operationId, reservation);
return reservation;
}
orderCreated(event) {
this.trace.emit(
'event-consumer',
'event',
`Notification получил event order.created для order=${event.orderId}`,
);
}
}
Dependencies(LAB_TRACE)(InventoryMessageController);
Controller()(InventoryMessageController);
const reserveDescriptor = Object.getOwnPropertyDescriptor(
InventoryMessageController.prototype,
'reserve',
);
MessagePattern({ cmd: 'inventory.reserve' })(
InventoryMessageController.prototype,
'reserve',
reserveDescriptor,
);
Payload()(
InventoryMessageController.prototype,
'reserve',
0,
);
const eventDescriptor = Object.getOwnPropertyDescriptor(
InventoryMessageController.prototype,
'orderCreated',
);
EventPattern('order.created')(
InventoryMessageController.prototype,
'orderCreated',
eventDescriptor,
);
Payload()(
InventoryMessageController.prototype,
'orderCreated',
0,
);
class MicroservicesLabModule {}
Module({
controllers: [InventoryMessageController],
})(MicroservicesLabModule);
export async function microservicesMessaging(emit) {
const port = await reserveTcpPort();
const module = {
module: MicroservicesLabModule,
providers: [
{
provide: LAB_TRACE,
useValue: { emit },
},
],
};
const options = {
transport: Transport.TCP,
options: {
host: '127.0.0.1',
port,
},
};
const service = await NestFactory.createMicroservice(module, {
...options,
logger: false,
});
const client = ClientProxyFactory.create(options);
try {
emit(
'boundary',
'start',
'Запускаем отдельную Nest application boundary с TCP transport',
);
await service.listen();
await client.connect();
const command = {
operationId: 'reserve-order-42',
orderId: 'order-42',
sku: 'node-book',
quantity: 1,
};
emit(
'producer',
'send',
'Checkout вызывает request-response pattern inventory.reserve и ждёт один ответ',
);
const first = await firstValueFrom(
client
.send({ cmd: 'inventory.reserve' }, command)
.pipe(timeout(2_000)),
);
emit(
'request-response',
'result',
`Получен reservation=${first.reservationId}; reused=${first.reused}`,
);
const duplicate = await firstValueFrom(
client
.send({ cmd: 'inventory.reserve' }, command)
.pipe(timeout(2_000)),
);
emit(
'idempotency',
'result',
`Повторный command не создал вторую запись; reused=${duplicate.reused}`,
);
emit(
'producer',
'emit',
'Checkout публикует event order.created и не ждёт business-ответ consumer-а',
);
await lastValueFrom(
client
.emit('order.created', { orderId: command.orderId })
.pipe(timeout(2_000)),
{ defaultValue: undefined },
);
try {
await firstValueFrom(
client
.send(
{ cmd: 'inventory.reserve' },
{
...command,
operationId: 'reserve-order-43',
orderId: 'order-43',
sku: 'sold-out',
},
)
.pipe(timeout(2_000)),
);
} catch (error) {
const code = error?.code ?? 'REMOTE_ERROR';
emit(
'remote-error',
'error',
`Remote error пересёк transport boundary: ${code}`,
);
}
emit(
'architecture',
'result',
'Микросервисы дают независимые границы deploy и владения данными, но добавляют сеть, partial failures, contracts, observability и delivery semantics',
);
} finally {
client.close();
await service.close();
emit('cleanup', 'done', 'TCP client и Nest microservice остановлены');
}
}
The emit(...) calls become live-trace rows. await and Promise keep the HTTP stream open until the scenario completes.
Practical patterns worth keeping nearby
Compare the goal, code, and caveats instead of memorizing syntax without a model.
Command consumer
Let Inventory answer a command and return a producer result.
@Controller()
export class InventoryMessages {
@MessagePattern({ cmd: 'inventory.reserve' })
reserve(@Payload() command: ReserveInventoryCommand) {
return this.inventory.reserve(command);
}
}- The pattern addresses a message inside the transport.
- @Payload extracts the deserialized body.
- The return value becomes the response.
Request-response producer
Send a command and wait for a bounded time.
const response = await firstValueFrom(
inventoryClient
.send(
{ cmd: 'inventory.reserve' },
{ operationId, orderId, sku, quantity },
)
.pipe(timeout(2_000)),
);- send creates an Observable request.
- firstValueFrom turns its first response into a Promise.
- timeout does not necessarily cancel the consumer.
Event consumer
Let Notifications react to an order that already exists.
@Controller()
export class OrderEvents {
@EventPattern('order.created.v1')
handle(@Payload() event: OrderCreatedV1) {
return this.notifications.sendConfirmation(event);
}
}- The name describes a completed fact.
- A version supports contract evolution.
- The publisher does not know every consumer.
Event publisher
Notify independent consumers after a successful commit.
this.events.emit('order.created.v1', {
eventId: randomUUID(),
occurredAt: new Date().toISOString(),
orderId,
customerId,
});- A production event is often published by an outbox relay.
- eventId supports deduplication and tracing.
Idempotent consumer
Avoid repeating a side effect after duplicate delivery.
await db.query(
`INSERT INTO processed_messages (consumer, message_id)
VALUES ($1, $2)
ON CONFLICT DO NOTHING
RETURNING message_id`,
['email-confirmation', event.eventId],
);- UNIQUE(consumer, message_id) protects races.
- Zero rowCount means duplicate.
- The marker and side effect need a consistent transaction boundary.
Transactional outbox
Avoid losing an event between database commit and broker publish.
await db.transaction(async (tx) => {
const order = await orders.insert(tx, input);
await outbox.insert(tx, {
type: 'order.created.v1',
aggregateId: order.id,
payload: { orderId: order.id },
});
});- The order and outbox row commit atomically.
- A relay retries broker publishing.
- Consumers still need duplicate safety.
How a learning mistake becomes an incident
A realistic service: the original code, observable failure, corrected implementation, and why the correction works.
Checkout creates a long synchronous chain across four services
A Nest CheckoutService creates an order and then sequentially waits for Inventory, Payments, and Notifications. Every remote call is required before HTTP 200.
Latencies add, availability multiplies, and a timeout after a successful payment leaves an unknown outcome. Retrying the whole HTTP request may charge or reserve twice.
@Injectable()
export class CheckoutService {
async checkout(input: CheckoutDto) {
const order = await this.orders.create(input);
await firstValueFrom(
this.inventory.send('reserve', order),
);
await firstValueFrom(
this.payments.send('charge', order),
);
await firstValueFrom(
this.notifications.send('email', order),
);
return { ...order, status: 'completed' };
}
}The HTTP request owns a distributed chain without one transaction manager. A successful remote side effect and a lost response are indistinguishable to the producer.
@Injectable()
export class CheckoutService {
async checkout(input: CheckoutDto) {
return this.db.transaction(async (tx) => {
const order = await this.orders.createPending(tx, input);
const eventId = randomUUID();
await this.outbox.add(tx, {
eventId,
type: 'order.placed.v1',
payload: { orderId: order.id },
});
return { orderId: order.id, status: 'pending' };
});
}
}
@Controller()
export class InventoryEvents {
@EventPattern('order.placed.v1')
handle(@Payload() event: OrderPlacedV1) {
return this.idempotency.once(
event.eventId,
() => this.inventory.reserve(event.orderId),
);
}
}The order and outbox event commit atomically. A relay retries publishing, while the consumer accepts possible duplicates through eventId. HTTP returns pending as the workflow converges asynchronously.
What the unfamiliar calls from both code samples actually do.
ClientProxy.send(pattern, data)- Creates a Nest request-response Observable whose producer expects one response or remote error.
firstValueFrom(observable)- Turns the first emitted RxJS response into a Promise suitable for await.
db.transaction(callback)- Runs order and outbox writes on one database connection with one COMMIT or ROLLBACK.
outbox.add(tx, event)- Stores a future event in the same transaction instead of trying to atomically coordinate PostgreSQL and a broker.
@EventPattern(name)- Registers a Nest consumer handler for an event pattern on the selected transport.
idempotency.once(eventId, work)- A representative application helper that atomically remembers eventId and skips work on duplicate delivery.
Common misconceptions
The myth is on the left; the accurate model is on the right.
Microservices automatically make a system scalable.
They permit independent scaling, but poor contracts and a shared database preserve coupling.
A service may synchronously call five more services.
Long request chains multiply latency and partial-failure probability.
A broker guarantees there are no duplicates.
Practical delivery usually requires idempotent consumers and deduplication.
A shared database is harmless convenience.
Direct writes into another service’s tables destroy ownership and independent deployment.
A pet project should begin with ten services.
A modular monolith is often cheaper until a measurable reason for extraction appears.
Explain it in your own words
If you can explain the answer without quoting documentation, your mental model is starting to take shape.
- When is request-response better than an event, and when is it worse?
- Why does a timeout not prove that remote work did not execute?
- Why do messages need eventId, operationId, and version?
- Why must an at-least-once consumer be idempotent?
- How does a transactional outbox close the dual-write gap?
- Why should two services not mutate the same business table?
- When is a modular monolith the more reasonable choice?