RRUNTIME LABruntime observatoryNNEON · Articles in 90 languages
CHAPTER 23
DEEP DIVE · FROM BASICS TO CODE

Understanding: Python objects, functions, and protocols

You can read this chapter before running the experiment. Then return to the live trace and match each concept to a real event.

FIRST, IN PLAIN LANGUAGE

Much of Python meaning lives in object protocols: a function is a value, class creates a new type, a generator pauses at yield, with guarantees context exit, and a decorator replaces a name with the result of another call.

TECHNICAL FOUNDATION

Functions, classes, and modules are objects. Attribute resolution follows the data model, bound methods receive the instance as self, exceptions unwind frames toward a matching handler, iter/next define iteration, and context managers implement enter/exit behavior.

Why it mattersThis is where JS intuition creates hidden bugs: a mutable default survives calls, class attributes are shared, shallow copies retain nested references, and generator bodies do not execute until iteration.
WHERE THE WORK RUNS
01NAMESbindings · scopes · closures
02OBJECTSidentity · type · value
03PROTOCOLSiteration · context · call
04APPLICATIONmodules · classes · errors
01 · GLOSSARY

Terms used in this experiment

Understand the words first, then the execution order.

01

Identity

The identity of one concrete object. is asks whether two references point to that same object.

02

Mutable

An object whose value can change without creating another object, such as list, dict, and set.

03

Callable

An object accepted by call syntax (). Functions and classes are callable, and custom objects may implement __call__.

04

Decorator

A callable that receives a function or class and binds the original name to its returned result.

05

Generator

An iterator that preserves a frame and resumes execution between yield expressions.

06

Context manager

An enter/exit protocol used by with for deterministic resource cleanup.

07

Exception

An error object propagated up call frames until a compatible except catches it.

08

dataclass

A standard-library decorator that generates common data-class methods such as __init__ and __repr__.

09

Closure

A function plus references to free variables from an enclosing function scope.

10

Module cache

sys.modules retains loaded modules, so repeated import normally returns the same module object.

02 · MECHANICS

What happens step by step

Each step maps to an observable runtime state.

  1. 01
    class executes its body

    Python evaluates the class body in a namespace and then creates a class object.

  2. 02
    Calling the class creates an instance

    __new__ creates the object and __init__ initializes the existing instance.

  3. 03
    A method binds self

    instance.method creates a bound method and passes the instance as the first argument.

  4. 04
    A decorator applies at definition time

    @decorator above def is equivalent to name = decorator(name).

  5. 05
    A generator starts on demand

    Calling its function creates a generator; the body begins only under next or iteration.

  6. 06
    with guarantees exit

    __exit__ runs for normal flow and exceptions, making cleanup predictable.

  7. 07
    An exception searches for a handler

    Frames unwind to a compatible except and finally runs on every exit path.

03 · CONTEXT

Where the result needs context

These details explain why similar code can sometimes produce a different trace.

01

Defaults are evaluated once

Default objects are created when def executes, so a mutable default accumulates state.

02

A class variable is shared

A mutable object in the class body is shared until an instance shadows that attribute.

03

property can run expensive code

order.total may invoke arbitrary descriptor code and raise an exception.

04

finally can replace a result

return inside finally suppresses an earlier result or exception and is usually a design bug.

05

A generator is single-use

After StopIteration the same generator does not restart; call the generator function again.

06

A closure retains objects

A captured object stays reachable while the inner function lives, just as in JavaScript.

07

Private is mainly convention

_name marks an internal API; __name uses name mangling but is not strict privacy.

01
Theory

First understand which parts of Node participate in execution.

02
Simplified code

Then remove instrumentation and focus on the central mechanism.

03
Runtime code

Finally match the model to the code that produces the live trace.

04 · Simplified code

A minimal model without instrumentation

src/python-lab.py · educational snippetPython
from dataclasses import dataclass

@dataclass(slots=True)
class CartLine:
    sku: str
    price: int
    quantity: int = 1

    @property
    def subtotal(self) -> int:
        return self.price * self.quantity

def even_squares(limit):
    for value in range(limit):
        if value % 2 == 0:
            yield value * value
05 · Runtime code

The complete code executed by the scenario

This is not an alternative example: these are the functions and files used by the Run button.

ACTUAL SOURCE

The first file is the real CPython scenario; the second safely starts the child process and turns its JSON Lines into the live trace.

src/python-lab.py
CPython scenario294 lines
#!/usr/bin/env python3
"""Fixed, input-free CPython scenarios used by Runtime Lab."""

from __future__ import annotations

import asyncio
import dis
import gc
import io
import json
import platform
import sys
import time
import weakref
from dataclasses import dataclass
from typing import Any, Iterator


def emit(lane: str, event_type: str, key: str, **data: Any) -> None:
    print(
        json.dumps(
            {"lane": lane, "type": event_type, "key": key, "data": data},
            ensure_ascii=False,
        ),
        flush=True,
    )


def version_event() -> None:
    emit(
        "python",
        "runtime",
        "python.version",
        implementation=platform.python_implementation(),
        version=platform.python_version(),
    )


def run_syntax() -> None:
    version_event()
    orders = [
        {"id": "A-10", "status": "paid", "price": 120, "qty": 2},
        {"id": "A-11", "status": "draft", "price": 80, "qty": 1},
        {"id": "A-12", "status": "paid", "price": 50, "qty": 3},
    ]
    emit("objects", "state", "syntax.objects", count=len(orders))

    paid = [order for order in orders if order["status"] == "paid"]
    total = sum(order["price"] * order["qty"] for order in paid)
    emit(
        "comprehension",
        "result",
        "syntax.comprehension",
        ids=", ".join(order["id"] for order in paid),
        total=total,
    )

    first, *middle, last = [order["id"] for order in orders]
    emit(
        "sequence",
        "result",
        "syntax.unpack",
        first=first,
        middle=middle,
        last=last,
    )

    labels = [f"{index}:{order['id']}" for index, order in enumerate(orders, start=1)]
    emit("loop", "result", "syntax.enumerate", labels=", ".join(labels))

    def format_order(order: dict[str, Any], *, currency: str = "RUB") -> str:
        return f"{order['id']} · {order['price'] * order['qty']} {currency}"

    emit(
        "function",
        "result",
        "syntax.function",
        rendered=format_order(orders[0], currency="₽"),
    )
    emit("result", "result", "syntax.result")


@dataclass(slots=True)
class CartLine:
    sku: str
    price: int
    quantity: int = 1

    @property
    def subtotal(self) -> int:
        return self.price * self.quantity


def append_bad(item: str, bucket: list[str] = []) -> list[str]:
    bucket.append(item)
    return list(bucket)


def append_safe(item: str, bucket: list[str] | None = None) -> list[str]:
    target = [] if bucket is None else bucket
    target.append(item)
    return target


def even_squares(limit: int) -> Iterator[int]:
    for value in range(limit):
        if value % 2 == 0:
            yield value * value


def classify_event(event: dict[str, Any]) -> str:
    match event:
        case {"type": "order.paid", "payload": {"id": order_id}}:
            return f"paid order {order_id}"
        case {"type": event_type}:
            return f"other event {event_type}"
        case _:
            return "invalid event"


def run_semantics() -> None:
    version_event()
    line = CartLine("book", 450, quantity=2)
    emit(
        "class",
        "result",
        "semantics.dataclass",
        rendered=repr(line),
        subtotal=line.subtotal,
    )

    original = ["node"]
    alias = original
    alias.append("python")
    emit("objects", "mutation", "semantics.alias", shared=original == alias == ["node", "python"])

    bad_first = append_bad("api")
    bad_second = append_bad("worker")
    emit(
        "function",
        "warning",
        "semantics.mutable-default",
        first=bad_first,
        second=bad_second,
    )

    safe_first = append_safe("api")
    safe_second = append_safe("worker")
    emit(
        "function",
        "result",
        "semantics.safe-default",
        first=safe_first,
        second=safe_second,
    )

    emit("generator", "result", "semantics.generator", values=list(even_squares(7)))
    emit(
        "pattern",
        "result",
        "semantics.match",
        label=classify_event({"type": "order.paid", "payload": {"id": "A-42"}}),
    )

    try:
        int("not-a-number")
    except ValueError as error:
        emit("exception", "caught", "semantics.exception", message=str(error))

    stream = io.StringIO()
    with stream:
        stream.write("cleanup is deterministic")
        text = stream.getvalue()
    emit("context", "cleanup", "semantics.context", closed=stream.closed, text=text)
    emit("result", "result", "semantics.result")


def doubled_total(values: list[int]) -> int:
    return sum(value * 2 for value in values)


class CycleNode:
    def __init__(self) -> None:
        self.peer: CycleNode | None = None


async def traced_task(name: str, delay: float, completion: list[str]) -> str:
    emit("asyncio", "start", "asyncio.started", name=name)
    await asyncio.sleep(delay)
    completion.append(name)
    emit("asyncio", "resume", "asyncio.resumed", name=name)
    return name


async def measure_timer_while(blocking_call: Any) -> float:
    started = time.perf_counter()
    timer = asyncio.create_task(asyncio.sleep(0.01))
    await blocking_call()
    await timer
    return max(0.0, (time.perf_counter() - started - 0.01) * 1_000)


async def run_asyncio_round() -> None:
    completion: list[str] = []
    first = asyncio.create_task(traced_task("A", 0.025, completion))
    second = asyncio.create_task(traced_task("B", 0.005, completion))
    emit("asyncio", "schedule", "asyncio.created")
    await asyncio.gather(first, second)
    emit("asyncio", "result", "asyncio.result", order=" → ".join(completion))

    async def block_loop() -> None:
        time.sleep(0.055)

    async def offload_sleep() -> None:
        await asyncio.to_thread(time.sleep, 0.055)

    blocked_delay = await measure_timer_while(block_loop)
    emit("asyncio", "blocking", "asyncio.blocking", delay=round(blocked_delay, 1))
    offloaded_delay = await measure_timer_while(offload_sleep)
    emit("asyncio", "offload", "asyncio.offload", delay=round(offloaded_delay, 1))


def run_runtime() -> None:
    implementation = platform.python_implementation()
    gil_probe = getattr(sys, "_is_gil_enabled", None)
    gil_enabled = gil_probe() if gil_probe else implementation == "CPython"
    emit(
        "runtime",
        "config",
        "runtime.config",
        implementation=implementation,
        version=platform.python_version(),
        gil=gil_enabled,
    )

    operations = [instruction.opname for instruction in dis.get_instructions(doubled_total)]
    emit(
        "bytecode",
        "result",
        "runtime.bytecode",
        operations=" → ".join(operations[:12]),
    )

    frame = sys._getframe()
    visible_locals = ", ".join(sorted(name for name in frame.f_locals if not name.startswith("_")))
    emit(
        "frame",
        "state",
        "runtime.frame",
        functionName=frame.f_code.co_name,
        locals=visible_locals,
    )

    left = CycleNode()
    right = CycleNode()
    left.peer = right
    right.peer = left
    left_ref = weakref.ref(left)
    right_ref = weakref.ref(right)
    alive_before = left_ref() is not None and right_ref() is not None
    del left, right
    collected = gc.collect()
    alive_after = left_ref() is not None or right_ref() is not None
    emit(
        "gc",
        "result",
        "runtime.gc",
        collected=collected,
        aliveBefore=alive_before,
        aliveAfter=alive_after,
    )

    asyncio.run(run_asyncio_round())
    emit("result", "result", "runtime.result")


SCENARIOS = {
    "syntax": run_syntax,
    "semantics": run_semantics,
    "runtime": run_runtime,
}


def main() -> None:
    scenario = sys.argv[1] if len(sys.argv) > 1 else ""
    runner = SCENARIOS.get(scenario)
    if runner is None:
        raise SystemExit(f"unknown scenario: {scenario}")
    runner()


if __name__ == "__main__":
    main()

The scenario instruments its own trace: Python calls emit(...) and prints ordered events as JSON Lines, while Node records a timestamp as each line arrives. Source and lane labels come from application code, not a CPython/asyncio profiler. The Node bridge reads output without a shell, bounds runtime and volume, and forwards events into the HTTP stream.

06 · RECIPES

Practical patterns worth keeping nearby

Compare the goal, code, and caveats instead of memorizing syntax without a model.

01

Function as a value

Recognize a callable without invoking it.

def normalize(value: str) -> str:
    return value.strip().lower()

pipeline = [normalize, str.upper]
for transform in pipeline:
    value = transform(value)
  • A function name without () refers to the function object.
  • Each callable receives the previous result.
02

Safe default

Avoid sharing mutable state.

def add_tag(tag: str, tags: list[str] | None = None):
    result = [] if tags is None else list(tags)
    result.append(tag)
    return result
  • None is an immutable sentinel.
  • list(tags) avoids mutating the caller list.
03

Data class

Read dataclass and property syntax.

from dataclasses import dataclass

@dataclass(frozen=True, slots=True)
class Money:
    amount: int
    currency: str = "USD"

    @property
    def label(self):
        return f"{self.amount} {self.currency}"
  • frozen is not deep immutability.
  • slots changes instance layout.
04

Generator

Process a stream lazily.

def read_valid(lines):
    for line in lines:
        text = line.strip()
        if text:
            yield text

for item in read_valid(file):
    consume(item)
  • The result is not stored in full.
  • The outer resource still needs cleanup.
05

Exception chaining

Add domain meaning without losing the cause.

try:
    user = repository.get(user_id)
except DatabaseError as error:
    raise UserLoadError(user_id) from error
else:
    return user
finally:
    metrics.increment("load.attempt")
  • raise ... from preserves the causal chain.
  • else runs only when try succeeds.
06

Context manager

Guarantee transaction cleanup.

with database.transaction() as transaction:
    order = transaction.insert_order(payload)
    transaction.insert_outbox(order)
  • The concrete manager defines commit and rollback behavior.
  • An exception is passed into __exit__.
07 · PRODUCTION CASES

How a learning mistake becomes an incident

A realistic service: the original code, observable failure, corrected implementation, and why the correction works.

CASE 01

Tags from one HTTP request appear in another

A helper adds audit tags and runs for every request. The author expects the default list to be created again on each function call.

INCIDENT CONTEXT

The default is evaluated once when def executes. A long-lived process reuses the list, mixing user metadata and gradually retaining more memory.

BEFOREPROBLEMATIC IMPLEMENTATION
def attach_tag(tag: str, tags: list[str] = []) -> list[str]:
    tags.append(tag)
    return tags

def audit_request(request):
    return attach_tag(f"user:{request.user_id}")

One function object stores one reference to the default list in __defaults__. Every call without tags mutates that same object.

AFTERCORRECTED IMPLEMENTATION
def attach_tag(
    tag: str,
    tags: list[str] | None = None,
) -> list[str]:
    result = [] if tags is None else list(tags)
    result.append(tag)
    return result

def audit_request(request):
    return attach_tag(f"user:{request.user_id}")

The immutable None sentinel is safe to store as a default. Each omitted value creates a new list, and a passed list is copied before mutation.

FUNCTIONS AND CONSTRUCTS

What the unfamiliar calls from both code samples actually do.

tags=[]
The default expression runs once when the function object is created, so calls share one mutable list.
function.__defaults__
The tuple of positional default objects where the retained reference to the shared list can be observed directly.
tags is None
Uses an immutable singleton to signal that the caller did not provide its own collection.
list(tags)
Creates a shallow copy of the iterable so the helper append does not mutate the caller-owned list.
result.append(tag)
Mutates that specific result list in place and returns None, so append is not assigned back to the variable.
WHY THE CORRECTION WORKS

Use mutable defaults only for an intentional function-level cache with a visible contract, synchronization, and limits. Ordinary parameters should use None or an immutable value.

WHAT PRODUCTION SHOWED
  • Audit records contain tags belonging to previous users.
  • The returned list grows with process uptime.
  • A restart temporarily removes the defect by resetting hidden state.
08 · DO NOT CONFUSE

Common misconceptions

The myth is on the left; the accurate model is on the right.

MYTH

items=[] creates a fresh list per call.

ACTUALLY

One default list is created when def executes and is reused.

MYTH

self is a special keyword.

ACTUALLY

It is the conventional name of the first instance-method parameter.

MYTH

yield permanently ends the function.

ACTUALLY

yield pauses its frame and the next next call resumes after it.

MYTH

except Exception catches absolutely everything.

ACTUALLY

System signals such as KeyboardInterrupt derive from BaseException instead.

MYTH

with only works with files.

ACTUALLY

The protocol also supports locks, transactions, tracing, and temporary state.

MYTH

Import merely declares a dependency.

ACTUALLY

The first import executes module top-level code and may perform I/O or mutate state.

09 · SELF-CHECK

Explain it in your own words

If you can explain the answer without quoting documentation, your mental model is starting to take shape.

  1. When is a default argument evaluated?
  2. Why can instances share a class-level list?
  3. When does a generator body begin?
  4. Which methods does with invoke?
  5. Why is a property not necessarily a cheap field?