# Comments as Design Documentation

Comments are one of the most debated topics in software engineering. Ousterhout argues that comments are not merely helpful -- they are essential design documentation that captures information that cannot be expressed in code. The belief that "good code is self-documenting" is partially true for implementation details, but dangerously wrong for abstractions, design decisions, and cross-cutting concerns.


## Table of Contents
1. [Why Comments Matter](#why-comments-matter)
2. [The Four Types of Comments](#the-four-types-of-comments)
3. [Comment-Driven Design](#comment-driven-design)
4. [The "Self-Documenting Code" Myth](#the-self-documenting-code-myth)
5. [Maintaining Comments](#maintaining-comments)
6. [Comments Anti-Patterns](#comments-anti-patterns)
7. [Summary](#summary)

---

## Why Comments Matter

Code tells you **what** the program does. Comments tell you:
- **Why** it does it that way
- **What** the abstraction promises (the contract)
- **What** assumptions the code makes
- **What** alternatives were considered and rejected
- **What** constraints link this code to other modules
- **What** is not obvious from reading the code

Without comments, this information exists only in the original developer's head. When that developer moves on, the information is lost. Future developers must reverse-engineer intent from implementation -- an error-prone process that leads to incorrect changes and accumulated complexity.

## The Four Types of Comments

### 1. Interface Comments

**Purpose:** Define the abstraction that a module, class, or function presents to its users.

**This is the most important type of comment.** Interface comments form the contract between a module and its callers. They should describe:
- What the function/method does (at an abstract level)
- What each parameter means and its constraints
- What the return value represents
- What side effects occur
- What exceptions can be thrown and under what conditions
- What the caller must ensure before calling (preconditions)
- What the caller can assume after the call (postconditions)

**Examples:**

```python
def find_nearest(target: Point, candidates: list[Point],
                 max_distance: float = inf) -> Point | None:
    """Find the candidate point closest to target.

    Returns the nearest point from candidates, or None if no candidate
    is within max_distance of target. If multiple candidates are
    equidistant, returns the one that appears first in the list.

    Args:
        target: The reference point to measure distances from.
        candidates: Points to search. Must not be empty.
        max_distance: Maximum Euclidean distance to consider.
            Points farther than this are ignored. Defaults to
            infinity (consider all points).

    Returns:
        The nearest Point, or None if all candidates exceed
        max_distance.

    Raises:
        ValueError: If candidates is empty.
    """
```

```java
/**
 * Acquire a database connection from the pool.
 *
 * Blocks until a connection is available or the timeout expires.
 * The returned connection is guaranteed to be valid (tested with
 * a lightweight query before returning). The caller MUST close
 * the connection when done, which returns it to the pool.
 *
 * @param timeout maximum time to wait for a connection
 * @return a valid, open database connection
 * @throws TimeoutException if no connection is available within timeout
 * @throws PoolExhaustedException if the pool is permanently full
 *     (all connections in use and at max capacity)
 */
public Connection acquire(Duration timeout)
```

**Key rules for interface comments:**
- Describe the abstraction, not the implementation
- If the comment mentions implementation details (algorithms, data structures, internal variables), it is too detailed
- A developer should be able to use the module correctly by reading only the interface comment, without reading any implementation code
- If you cannot write a clear interface comment, the interface may be poorly designed

### 2. Data Structure Member Comments

**Purpose:** Explain the meaning, constraints, and invariants of fields in a class or data structure.

Field names alone rarely convey all the information a developer needs. Comments should clarify:
- What the field represents (especially if the name is ambiguous)
- Units and encoding (milliseconds? seconds? UTC? local time?)
- Valid ranges and boundary conditions
- Relationships with other fields
- When the field is set and when it may be null/zero

**Examples:**

```python
class RetryConfig:
    # Maximum number of retry attempts before giving up.
    # Does not count the initial attempt, so total attempts = max_retries + 1.
    # Set to 0 to disable retries.
    max_retries: int

    # Base delay between retries in milliseconds.
    # Actual delay uses exponential backoff: base_delay_ms * 2^attempt.
    # Jitter of +/- 20% is applied to prevent thundering herd.
    base_delay_ms: int

    # Maximum delay cap in milliseconds. Exponential backoff will
    # not exceed this value regardless of attempt number.
    # Must be >= base_delay_ms.
    max_delay_ms: int
```

```java
class PageCache {
    // Maps page_id to cached page content. Entries are evicted
    // in LRU order when the cache exceeds maxEntries. A page
    // present in this map is guaranteed to match the on-disk
    // version as of the last sync (see lastSyncTime).
    private Map<Long, Page> cache;

    // Timestamp of the last cache synchronization with disk,
    // in epoch milliseconds (UTC). All cache entries are valid
    // as of this time. Writes after this time may not be reflected.
    private long lastSyncTime;

    // Upper bound on cache entries. When exceeded, the least
    // recently accessed entry is evicted before inserting a new one.
    // Invariant: cache.size() <= maxEntries at all times.
    private int maxEntries;
}
```

### 3. Implementation Comments

**Purpose:** Explain **why** the code does something a particular way, or clarify non-obvious logic.

Implementation comments should not describe **what** the code does -- that should be clear from reading the code itself. They should explain:
- Why this approach was chosen over alternatives
- What non-obvious constraint or edge case the code handles
- What would go wrong if the code were changed in an obvious-seeming way
- Performance considerations that drove the implementation choice

**Good implementation comments:**

```python
# Use binary search instead of linear scan because the list is sorted
# and can contain 100k+ entries. Linear scan caused 200ms latency
# in production (see incident #4521).
index = bisect.bisect_left(sorted_entries, target)
```

```python
# Process items in reverse order to avoid index invalidation when
# removing elements. Forward iteration would skip elements after
# each removal.
for i in range(len(items) - 1, -1, -1):
    if should_remove(items[i]):
        items.pop(i)
```

```python
# Intentionally catching broad Exception here because the third-party
# library can throw undocumented exceptions (observed RuntimeError,
# ValueError, and OSError in production). We log and continue rather
# than crash the batch job.
try:
    result = third_party_lib.process(data)
except Exception as e:
    logger.warning(f"Processing failed for {data.id}: {e}")
    result = default_result()
```

**Bad implementation comments (just repeat the code):**

```python
# Increment counter
counter += 1

# Check if user is active
if user.is_active:

# Loop through items
for item in items:

# Return the result
return result
```

These comments add no information. The code already says what it does. Remove them.

### 4. Cross-Module Comments

**Purpose:** Document dependencies and design decisions that span multiple modules.

These are the hardest comments to maintain but often the most critical, because cross-module relationships are the biggest source of unknown unknowns.

**Examples:**

```python
# This timeout value must be longer than the retry timeout in
# RetryPolicy (currently 30s with 3 retries = 90s max). If this
# timeout is shorter, the caller will give up before retries complete.
# See: src/retry/policy.py:RetryPolicy.MAX_TOTAL_DURATION
REQUEST_TIMEOUT_SECONDS = 120
```

```python
# The field order in this struct must match the binary protocol
# defined in docs/protocol-v3.md section 4.2. The client parser
# (client/src/parser.rs) reads fields in this exact order.
# Changing field order here requires updating both the docs and
# the client parser.
class ServerMessage:
    version: int      # 2 bytes, big-endian
    message_type: int # 1 byte
    payload_len: int  # 4 bytes, big-endian
    payload: bytes    # payload_len bytes
```

```java
/**
 * IMPORTANT: This method is called by the EventBus on a background
 * thread. It must not access the UI thread directly. Use
 * Platform.runLater() for any UI updates.
 *
 * The EventBus guarantees at-least-once delivery, so this handler
 * must be idempotent. See EventBus.subscribe() docs for details.
 */
public void onOrderCompleted(OrderCompletedEvent event) {
```

**Best practices for cross-module comments:**
- Place the comment in the most likely place a developer would look
- Reference the other module explicitly (file path, class name)
- Explain what would go wrong if the relationship were violated
- Consider using a shared constants file for values that must stay in sync

## Comment-Driven Design

**Write the comments before writing the code.**

This is one of Ousterhout's most practical recommendations. The process:

1. **Write the interface comment first:** Before writing any implementation, write the comment that describes what the function/class/module does, what its parameters mean, and what it returns.

2. **Evaluate the design:** If the interface comment is hard to write, unclear, or requires mentioning implementation details, the interface design is probably wrong. Redesign the interface until the comment is clean and simple.

3. **Write the implementation:** With a clear interface comment as your guide, the implementation has a clear target.

4. **Add implementation comments:** As you write code, add comments for any non-obvious decisions.

### Why Comment-Driven Design Works

| Benefit | Explanation |
|---------|-------------|
| Forces clear thinking | Writing what something does before how reveals confusion early |
| Catches bad abstractions | If you can't describe the interface simply, it's too complex |
| Produces better interfaces | The act of writing clarifies what callers actually need |
| Comments stay accurate | Written alongside the design, not retrofitted later |
| Saves time | Avoids implementing a design that turns out to be wrong |

### Example

**Step 1:** Write the interface comment.

```python
def merge_sorted_streams(*streams: Iterator[T],
                          key: Callable = None) -> Iterator[T]:
    """Merge multiple sorted iterators into a single sorted iterator.

    Each input stream must be sorted in ascending order (or by key
    if provided). The output yields all elements from all streams
    in globally sorted order. Memory usage is O(num_streams),
    regardless of stream length.

    Equal elements are yielded in the order their source streams
    appear in the arguments (stable merge).
    """
```

**Step 2:** Evaluate. Is this clear? Can a caller use this without reading the implementation? What about edge cases -- empty streams, single stream, duplicate elements? Add those details if needed.

**Step 3:** Implement. The comment now serves as the specification.

## The "Self-Documenting Code" Myth

The claim that "good code doesn't need comments" contains a kernel of truth but is dangerously incomplete.

### Where Self-Documenting Code Works

Code **can** document itself for low-level implementation details:

```python
# This is self-documenting -- no comment needed:
total_price = sum(item.price for item in cart.items)
is_eligible = user.age >= 18 and user.has_valid_id
filtered = [x for x in data if x.is_active and x.score > threshold]
```

Good variable names, clear control flow, and simple expressions make the **what** obvious. Comments that restate this are noise.

### Where Self-Documenting Code Fails

Code **cannot** document:

| Information | Why Code Can't Express It | Example |
|------------|--------------------------|---------|
| **Abstractions** | Code shows implementation, not the promise | An interface's contract and guarantees |
| **Why** | Code shows what happens, not why this approach | Why binary search instead of hash lookup |
| **Constraints** | Code enforces constraints but doesn't explain them | Why a timeout is set to 120 seconds |
| **Design alternatives** | Code shows the choice made, not choices rejected | Why we chose polling over webhooks |
| **Cross-module relationships** | Code in one module can't describe its relationship to another | This timeout must match the retry config |
| **Performance rationale** | Optimized code is often less readable | Why we denormalized this data structure |
| **Assumptions** | Code operates on assumptions it cannot state | "This list is always sorted by the caller" |

### The Practical Rule

**Use self-documenting code for the "what" (implementation). Use comments for the "why" (design decisions), the "what" at a higher level (abstractions/interfaces), and the "beware" (non-obvious constraints and relationships).**

## Maintaining Comments

Comments that are wrong are worse than no comments. Here are strategies for keeping them accurate:

### 1. Place Comments Near the Code

The closer a comment is to the code it describes, the more likely it will be updated when the code changes. Interface comments in the function signature are better than comments in a separate documentation file.

### 2. Avoid Duplicating Information

If the same information is stated in a comment and enforced in code, one will eventually become stale. State each fact once.

```python
# Bad: duplicates the type annotation
# max_retries is an integer representing the maximum number of retries
max_retries: int  # The type already says it's an int

# Good: adds information not in the code
# Set to 0 to disable retries. Values > 10 are capped at 10 to prevent
# excessive load on the downstream service during outages.
max_retries: int
```

### 3. Update Comments in the Same Commit

Make it a code review norm: if you change a function's behavior, you must update its interface comment in the same commit. Stale comments are a code review finding.

### 4. Use Comments as a Design Smell Detector

If a comment is hard to write, the code may be too complex. If a comment needs to be very long, the interface may be doing too much. If a comment keeps going out of date, the module's boundaries may be wrong. Difficult comments are a signal, not just a chore.

### 5. Treat Comment Quality as a Review Criterion

In code reviews, evaluate comments alongside code:
- Are interface comments complete and accurate?
- Do implementation comments explain why, not what?
- Are cross-module comments present where needed?
- Are there missing comments on non-obvious code?

## Comments Anti-Patterns

| Anti-Pattern | Problem | Fix |
|-------------|---------|-----|
| **Comment repeats the code** | Adds noise, no information | Delete it; let the code speak for implementation details |
| **Comment describes what, not why** | Misses the valuable information | Rewrite to explain the reasoning or design decision |
| **Comment on every line** | Obscures code, hard to maintain | Comment only non-obvious sections; trust clear code |
| **TODO without context** | "TODO: fix this" is useless months later | Include the issue number, the problem, and the fix direction |
| **Commented-out code** | Dead code that confuses readers | Delete it; version control preserves history |
| **Banner comments** | `/////// SECTION ///////` adds structure without information | Use meaningful function/class boundaries instead |
| **Apology comments** | "Sorry, this is a hack" acknowledges but doesn't fix | Fix the hack or add context on why it is necessary and when it can be fixed |
| **Stale comments** | Describe behavior that no longer exists | Update or remove in the same commit as the code change |

## Summary

Comments are not a sign of bad code. They are design documentation that captures the most valuable and perishable information in a system: the designer's intent, the abstraction's contract, and the non-obvious relationships between components. Write interface comments first, maintain them alongside code, and use them as a tool for thinking clearly about design.

# Complexity: Symptoms, Causes, and Measurement

The single greatest challenge in software engineering is managing complexity. This reference details how to recognize complexity, understand its causes, and measure it informally to guide design decisions.

## Definition of Complexity

Complexity is anything related to the structure of a software system that makes it hard to understand and modify. It is not about the size of the system or the sophistication of its features. A large system with clean abstractions can be less complex than a small system with tangled dependencies.

Ousterhout defines it with a practical formula:

```
C = sum(cp * tp) for each part p
```

Where:
- `cp` is the complexity of part `p`
- `tp` is the fraction of time developers spend working on part `p`

A module that is extremely complex but never touched contributes little overall complexity. A module that is moderately complex but modified constantly dominates the system's effective complexity.

## The Three Symptoms of Complexity

### 1. Change Amplification

**Definition:** A seemingly simple change requires modifications in many different places.

**How to recognize it:**
- Adding a new field to a data model requires changes in 8+ files
- Changing a color scheme means updating dozens of components
- Adding a new API endpoint requires modifications in routing, validation, serialization, testing, and documentation files that all repeat similar patterns

**Examples:**

| Symptom | Root Cause | Better Design |
|---------|-----------|---------------|
| Adding a database column touches 12 files | Schema knowledge is scattered across ORM, API, serialization, validation layers | Use a single source of truth for schema that generates other artifacts |
| Changing error message format requires editing every handler | Error formatting is duplicated in each endpoint | Centralize error formatting in middleware |
| Adding a new event type requires changes in producer, consumer, schema, and 3 processors | Event structure knowledge is not encapsulated | Define event schemas in one place; processors discover structure from schema |
| Renaming a field touches API, database, frontend, tests | The same concept is named differently in each layer | Use consistent naming conventions and code generation where possible |

**The test:** Ask "If I need to make this change, how many files do I need to touch?" If the answer is more than 2-3 for a conceptually simple change, you have change amplification.

### 2. Cognitive Load

**Definition:** A developer must know too much to complete a task safely.

**How to recognize it:**
- You need to read 5 files to understand what one function does
- A function has 8 parameters, each with non-obvious constraints
- Understanding the order of operations requires knowing implementation details of 3 other modules
- Global state means any function could have side effects that affect your code

**Examples:**

| Symptom | Root Cause | Better Design |
|---------|-----------|---------------|
| Must understand memory allocation to use an API | Interface leaks implementation details | Hide allocation behind the API; manage memory internally |
| Must configure 6 parameters before calling a function | Module pushes decisions to callers | Provide sensible defaults; auto-detect where possible |
| Must hold 4 invariants in mind when modifying a data structure | Invariants are not enforced by the module | Encapsulate invariants inside the module; enforce them automatically |
| Must read all callers before changing a shared utility | Utility has implicit contracts with each caller | Define explicit interfaces; use type systems to enforce contracts |

**The test:** Ask "How much does a developer need to know to use this module correctly?" If the answer involves understanding the implementation, the interface is too complex.

**Important nuance:** Lines of code can be misleading. An approach with more lines but less cognitive load is preferable. A 10-line function that requires understanding 5 external systems is more complex than a 30-line function that is self-contained.

### 3. Unknown Unknowns

**Definition:** It is not obvious which pieces of code must be changed, or what information is needed to make a change. This is the worst symptom because you don't even know what you don't know.

**How to recognize it:**
- A change seems to work in testing but breaks something unrelated in production
- A developer makes a reasonable change but violates an undocumented assumption
- The only way to learn about a constraint is to break it
- Knowledge exists only in one developer's head

**Examples:**

| Symptom | Root Cause | Better Design |
|---------|-----------|---------------|
| Changing module A breaks module C through a hidden dependency via B | Implicit dependency chain | Make dependencies explicit through interfaces and type systems |
| A race condition only surfaces under load | Concurrency assumptions are undocumented | Document threading model; use constructs that make concurrency visible |
| Reordering initialization steps causes silent data corruption | Initialization order dependency is implicit | Make ordering explicit through dependency injection or builder patterns |
| Modifying a "private" helper breaks an external system that depends on its behavior | Internal implementation has undocumented external consumers | Define clear public APIs; use access control to enforce boundaries |

**The test:** Ask "Can a new developer make changes to this module confidently without talking to someone?" If the answer is no, you have unknown unknowns.

## The Two Causes of Complexity

### Dependencies

A dependency exists when code cannot be understood or modified in isolation -- the code relates to other code in some way.

**Types of dependencies:**

| Type | Description | Example |
|------|-------------|---------|
| **Syntactic** | Compiler/linter will catch if broken | Function signature changes; import errors |
| **Semantic** | Compiler cannot catch; behavior depends on understanding | Two modules must agree on a data format not enforced by types |
| **Temporal** | Code must execute in a specific order | Init must happen before use; close must happen after all writes |
| **Hidden** | No visible indication of the relationship | Module A's behavior depends on global state set by module B |

**Goal:** You cannot eliminate dependencies entirely (software is interconnected), but you can:
1. Minimize the number of dependencies
2. Make remaining dependencies obvious and simple
3. Prefer syntactic dependencies over semantic ones (the compiler helps you)

### Obscurity

Obscurity occurs when important information is not obvious.

**Common sources:**

- Generic variable names: `data`, `temp`, `result`, `info`, `manager`
- Inconsistent naming: the same concept called `user` in one module and `account` in another
- Missing documentation: no explanation of why a design decision was made
- Non-obvious side effects: a function named `getUser()` that also updates a cache
- Magic numbers: `if retries > 3` without explaining why 3
- Implicit conventions: "All timestamps are UTC" but it is never stated

**The fix:** Make things obvious through:
1. Precise naming that conveys meaning
2. Comments that explain why, not what
3. Consistent conventions applied everywhere
4. Type systems that encode constraints
5. Explicit rather than implicit behavior

## Complexity Is Incremental

This is one of Ousterhout's most important observations: complexity rarely arrives as a single large problem. Instead, it accumulates from hundreds of small decisions.

**The pattern:**
1. A developer takes a small shortcut: "This one special case won't matter"
2. Another developer adds a small workaround: "It's just one extra parameter"
3. A third developer duplicates some logic: "Refactoring would take too long right now"
4. After a year, the system is difficult to work with, but no single change caused it

**Why this matters:**
- There is no single big fix for incremental complexity
- You cannot "refactor away" complexity in a weekend -- it must be managed continuously
- Every small decision matters: each shortcut contributes its small fraction
- The "broken windows" effect applies: once a module is messy, developers stop trying to keep it clean

**The discipline:**
- Adopt a zero-tolerance policy for complexity growth
- Every PR should leave the code at least as clean as it found it
- Small design improvements in every change compound into a great codebase over time
- Think of complexity like financial debt: each shortcut is a small loan with interest

## Measuring Complexity Informally

There is no precise metric for complexity, but you can measure it through proxies:

### Developer Experience Questions

| Question | Good Answer | Bad Answer |
|----------|------------|------------|
| "How long does it take a new team member to make their first meaningful change?" | Days | Weeks or months |
| "When you make a change, how confident are you that nothing else breaks?" | Very confident | Nervous; need extensive testing |
| "How many files do you typically touch for a feature?" | 1-3 | 5+ |
| "Can you explain what module X does in one sentence?" | Yes, clearly | It does... a lot of things |
| "When was the last time a change had unexpected side effects?" | Rarely | Last week |

### Code-Level Signals

| Signal | Low Complexity | High Complexity |
|--------|---------------|-----------------|
| Interface size (parameters, methods) | Few, cohesive | Many, unrelated |
| Module size | Varies (depth matters more) | Very large with intertwined concerns |
| Change locality | Changes are local to 1-2 modules | Changes ripple across many modules |
| Test fragility | Tests break only when behavior changes | Tests break when implementations change |
| Onboarding time | New developers productive in days | New developers need weeks of mentoring |

### The "What Is the Simplest Interface?" Test

For any module, ask: "What is the simplest interface that would meet all the current use cases?"

Compare the current interface to this ideal:
- If they match, the module is well-designed
- If the current interface is significantly more complex, there is unnecessary complexity
- If you cannot define a simple interface, the module may be doing too much

## Red Flags for Complexity

Ousterhout identifies several "red flags" -- patterns that signal complexity problems:

| Red Flag | What It Signals |
|----------|----------------|
| Shallow module | Interface is not much simpler than implementation |
| Information leakage | Same knowledge in multiple modules |
| Temporal decomposition | Modules split by time rather than knowledge |
| Overexposure | API exposes internal state that callers shouldn't need |
| Pass-through method | Method does nothing except call another method with same arguments |
| Repetition | Same code pattern appears in multiple places |
| Special-general mixture | General-purpose module has special-case code for specific callers |
| Conjoined methods | You can't understand method A without reading method B |
| Comment repeats code | Comment says the same thing as the code, adding no information |
| Vague name | Name does not convey what the thing does |

## Applying the Framework

When designing a new module or reviewing existing code:

1. **Identify the symptoms:** Is there change amplification? Cognitive load? Unknown unknowns?
2. **Trace the causes:** Are there unnecessary dependencies? Is important information obscure?
3. **Apply the simplest interface test:** What is the simplest interface that meets current needs?
4. **Check for red flags:** Does the design exhibit any of the patterns above?
5. **Decide on action:** Does the complexity warrant a redesign, or is it manageable?

The goal is not perfection but continuous improvement. Each design decision that reduces complexity, even slightly, contributes to a system that remains manageable over time.

# Deep vs Shallow Modules

The concept of module depth is one of the most powerful ideas in Ousterhout's philosophy. It provides a concrete way to evaluate whether a module is pulling its weight in the system.


## Table of Contents
1. [The Core Idea](#the-core-idea)
2. [Visualizing Module Depth](#visualizing-module-depth)
3. [Examples of Deep Modules](#examples-of-deep-modules)
4. [Examples of Shallow Modules](#examples-of-shallow-modules)
5. [The Disease of Classitis](#the-disease-of-classitis)
6. [When Shallow Is Acceptable](#when-shallow-is-acceptable)
7. [Designing for Depth](#designing-for-depth)
8. [Measuring Depth in Practice](#measuring-depth-in-practice)
9. [Common Objections](#common-objections)

---

## The Core Idea

Every module has two parts:
- **Interface:** The complexity it imposes on the rest of the system (the cost)
- **Implementation:** The functionality it provides (the benefit)

A module's value is determined by the ratio of functionality provided to interface complexity imposed.

```
Module Value = Functionality / Interface Complexity
```

**Deep modules** have high value: they provide a lot of functionality through a simple interface. **Shallow modules** have low value: their interface is nearly as complex as their implementation, so they add little net simplification to the system.

## Visualizing Module Depth

Think of a module as a rectangle:
- Width at the top = interface complexity
- Height = implementation depth (functionality hidden)

```
Deep Module:               Shallow Module:
┌──────┐                   ┌──────────────────────┐
│      │                   │                      │
│      │                   └──────────────────────┘
│      │
│      │
│      │
│      │
└──────┘
Narrow interface,          Wide interface,
deep implementation.       shallow implementation.
```

The goal is tall, narrow rectangles: modules that hide substantial complexity behind small interfaces.

## Examples of Deep Modules

### Unix File I/O

The Unix file I/O interface is one of the deepest abstractions in computing:

```c
int open(const char *path, int flags);
int close(int fd);
ssize_t read(int fd, void *buf, size_t count);
ssize_t write(int fd, const void *buf, size_t count);
off_t lseek(int fd, off_t offset, int whence);
```

Five functions. Behind this simple interface, the implementation handles:
- Disk block allocation and management
- Directory traversal and path resolution
- File permissions and access control
- Buffer caching and write-back strategies
- Device driver communication
- File system journal and crash recovery
- Network file system protocols (NFS)
- Memory-mapped file coordination
- Concurrent access and locking

The interface is measured in a few functions; the implementation is hundreds of thousands of lines of code. This is extreme depth.

### Garbage Collectors

A garbage collector's interface is essentially invisible:

```
Interface: (none -- just allocate objects normally)
```

Behind this zero-complexity interface, the implementation handles:
- Reference tracking and reachability analysis
- Generational collection strategies
- Compaction and memory defragmentation
- Concurrent collection without stopping the world
- Weak references and finalization
- Heap sizing and growth heuristics

The deepest modules are those whose interfaces are so simple that callers may not even realize they exist.

### TCP/IP Networking

```python
socket.send(data)
socket.recv(buffer_size)
```

Behind this:
- Packet segmentation and reassembly
- Retransmission and acknowledgment
- Flow control and congestion avoidance
- Routing across networks
- Checksum verification
- Connection state management
- Out-of-order packet handling

### Hash Maps

```python
map[key] = value
value = map[key]
del map[key]
```

Behind this:
- Hash function computation
- Collision resolution (chaining, open addressing)
- Dynamic resizing and rehashing
- Memory allocation strategies
- Load factor management
- Iterator invalidation handling

## Examples of Shallow Modules

### Java I/O Classes (Classic Example)

To read a serialized object from a file in Java:

```java
FileInputStream fileStream = new FileInputStream(filename);
BufferedInputStream bufferedStream = new BufferedInputStream(fileStream);
ObjectInputStream objectStream = new ObjectInputStream(bufferedStream);
```

Three classes, each adding a thin layer:
- `FileInputStream`: reads bytes from a file (no buffering)
- `BufferedInputStream`: adds buffering (why isn't this default?)
- `ObjectInputStream`: deserializes objects

Each class is shallow: its interface is nearly as complex as its implementation. The total cognitive load of three interfaces is greater than what a single deep interface would impose. A deep design would look like:

```java
ObjectInputStream stream = new ObjectInputStream(filename);
// Handles file opening, buffering, and deserialization internally
```

### Thin Wrapper Classes

```python
class UserValidator:
    def validate(self, user):
        if not user.name:
            raise ValueError("Name required")
        if not user.email:
            raise ValueError("Email required")

class UserSaver:
    def save(self, user):
        self.db.insert(user)

class UserService:
    def create_user(self, data):
        user = User(data)
        self.validator.validate(user)
        self.saver.save(user)
```

Three classes where one would suffice:

```python
class UserService:
    def create_user(self, data):
        user = User(data)
        if not user.name:
            raise ValueError("Name required")
        if not user.email:
            raise ValueError("Email required")
        self.db.insert(user)
```

The three-class version creates two additional interfaces (and their tests, files, and import chains) without providing meaningful abstraction. The validation and persistence logic is too simple to justify separate modules.

### Pass-Through Methods

```python
class OrderController:
    def create_order(self, request):
        order_data = self.parse_request(request)
        return self.order_service.create_order(order_data)

class OrderService:
    def create_order(self, order_data):
        validated = self.validate(order_data)
        return self.order_repository.create_order(validated)

class OrderRepository:
    def create_order(self, order_data):
        return self.db.insert("orders", order_data)
```

Each layer adds almost nothing. The `create_order` method appears three times, each just passing data to the next layer. This is a sign of shallow decomposition.

## The Disease of Classitis

**Classitis** is the misguided belief that "classes should be small" applied without judgment. It produces systems with hundreds of tiny classes, each doing very little, connected by a web of interfaces.

### Symptoms of Classitis

| Symptom | Example |
|---------|---------|
| Many classes with 10-30 lines each | `StringHelper`, `DateFormatter`, `NullChecker` |
| Most methods are one-liners or delegates | `getName() { return this.name; }` |
| Understanding a feature requires reading 8+ classes | Controller, Service, Repository, Mapper, Validator, DTO, Entity, Factory |
| Class names end in -Helper, -Util, -Manager, -Handler | `UserManager`, `OrderHandler`, `DataHelper` |
| Most classes have only 1-2 methods | A `Validator` class with only `validate()` |

### Why Classitis Happens

1. **Misinterpreted "Single Responsibility Principle"**: SRP says "one reason to change," not "one thing it does." A module can do many things if they all change together.
2. **Cargo cult patterns**: Applying patterns (Strategy, Factory, Builder) reflexively without evaluating whether they add depth.
3. **Metrics worship**: Optimizing for "small class size" or "few methods per class" instead of depth.
4. **Test-driven granularity**: Creating classes just to make them independently testable, even when they have no independent meaning.

### The Cure

Ask for each class: **"Does this class hide significant complexity behind its interface?"**

If the answer is no, it is a candidate for merging with another class. Fewer, deeper classes almost always produce simpler systems than many shallow ones.

## When Shallow Is Acceptable

Not every module needs to be deep. Shallow modules are acceptable when:

| Situation | Why It's OK | Example |
|-----------|------------|---------|
| **Dispatchers** | Routing logic is inherently shallow | A URL router that maps paths to handlers |
| **Interface adapters** | Translating between two deep modules | Converting between internal and external data formats |
| **Language/framework requirements** | The framework demands the class | Java servlets, Python ABC implementations |
| **Genuine one-liner utilities** | The abstraction is the name itself | `isEven(n)`, `clamp(value, min, max)` |
| **Entry points** | Top-level wiring that connects modules | The `main()` function, dependency injection configuration |

The key is that these shallow modules should be **rare exceptions**, not the norm. If most of your modules are shallow, the design needs rethinking.

## Designing for Depth

### Strategy 1: Combine Related Functionality

Instead of:
```
RequestParser + RequestValidator + RequestAuthorizer + RequestHandler + ResponseBuilder
```

Consider:
```
RequestHandler (parses, validates, authorizes, handles, and builds response)
```

If these operations always happen together and share knowledge about the request format, combining them into one deep module eliminates four interfaces and produces a simpler system.

### Strategy 2: Hide Implementation Decisions

Ask: "What decisions does this module make that no one else needs to know about?"

Each hidden decision adds depth. Good examples:
- Buffer sizes and caching strategies
- Retry logic and backoff policies
- Connection pooling and lifecycle management
- Data format and serialization details
- Concurrency and locking strategies

### Strategy 3: Provide Defaults

Instead of requiring callers to specify everything:

```python
# Shallow: caller must know about all options
def connect(host, port, timeout, retry_count, retry_delay,
            ssl_cert, ssl_key, keepalive, buffer_size):

# Deep: sensible defaults hide decisions
def connect(host, port=5432, **options):
    # Internally determines timeout, retries, SSL, etc.
```

### Strategy 4: Absorb Complexity

When two approaches exist -- one that is simpler for the module but pushes complexity to callers, and one that is harder to implement but simpler for callers -- choose the one that makes life easier for callers.

```python
# Pushes complexity to caller:
entries = log.read_raw()  # Returns raw bytes; caller must parse
parsed = parse_log_format(entries)  # Caller needs format knowledge

# Absorbs complexity:
entries = log.read()  # Returns parsed, structured entries
```

### Strategy 5: Question Every Interface Element

For each method, parameter, or return value in an interface, ask:
- "Do callers actually need this?"
- "Can the module decide this internally?"
- "Is there a simpler way to express this?"

Remove anything that does not earn its place. Every element in an interface is a cost that must be justified by the functionality it enables.

## Measuring Depth in Practice

### Quick Assessment

| Question | Deep | Shallow |
|----------|------|---------|
| How many methods in the interface? | Few (3-7) | Many (15+) |
| How many parameters per method? | Few (1-3) | Many (5+) |
| How long is the implementation? | Significantly larger than interface | About the same as interface |
| Can you describe the module in one sentence? | Yes | Need a paragraph |
| Does the module hide a non-trivial decision? | Yes, several | Not really |
| Would removing it require callers to duplicate code? | Lots of duplication | Minimal duplication |

### Depth Ratio

A rough heuristic: compare the lines of interface documentation to lines of implementation. If they are close to equal, the module is likely shallow. If the implementation is 5-10x larger than the interface description, the module is likely deep.

This is not about lines of code per se -- it is about the amount of hidden complexity relative to the exposed interface. A one-line interface like `gc.collect()` that hides thousands of lines of garbage collection logic is extremely deep.

## Common Objections

### "But small classes are easier to test!"

Small classes are easier to **unit test** in isolation, but the system is harder to **integration test** because you have more interfaces to mock, more interactions to verify, and more wiring to get right. Deeper modules that own more behavior are often easier to test at the level that matters: "does this feature work?"

### "But the Single Responsibility Principle says..."

SRP says a module should have "one reason to change," which is about **cohesion**, not about size. A module that handles all aspects of file I/O (opening, reading, writing, buffering, closing) changes for one reason: when file I/O requirements change. That is a single responsibility implemented deeply.

### "But what about separation of concerns?"

Separation of concerns is about keeping unrelated things apart, not about splitting related things into tiny pieces. If parsing, validating, and processing a request are all concerned with "handling a request," they can live in one module. Separate concerns that are genuinely independent (e.g., logging and business logic), not every step of a single workflow.

# General-Purpose vs Special-Purpose Modules

One of the most important design decisions is how general-purpose or special-purpose a module's interface should be. Ousterhout advocates for a "somewhat general-purpose" approach: general enough to avoid special cases, specific enough to avoid over-engineering.


## Table of Contents
1. [The Spectrum](#the-spectrum)
2. [The Key Question](#the-key-question)
3. [Push Complexity Downward](#push-complexity-downward)
4. [Configuration Parameters: Complexity Amplifiers](#configuration-parameters-complexity-amplifiers)
5. [When Specialization Is Justified](#when-specialization-is-justified)
6. [Practical Guidelines](#practical-guidelines)
7. [The Relationship to Information Hiding](#the-relationship-to-information-hiding)
8. [Summary](#summary)

---

## The Spectrum

```
Too Special ←————————————————————————→ Too General
  (bloated     (sweet spot:              (wasted effort,
  with         "somewhat                  unnecessary
  special      general-purpose")          abstraction)
  cases)
```

### Too Special-Purpose

A module designed for one specific use case. Its interface includes details that tie it to a particular caller or scenario.

```python
# Too special: designed for one specific email use case
class WelcomeEmailSender:
    def send_welcome_email(self, user_name, user_email, plan_name):
        ...

class PasswordResetEmailSender:
    def send_reset_email(self, user_email, reset_token, expiry_minutes):
        ...

class InvoiceEmailSender:
    def send_invoice_email(self, user_email, invoice_id, amount, due_date):
        ...
```

Three classes doing essentially the same thing (sending email) with interfaces tied to specific use cases. Adding a fourth email type requires creating another class.

### Too General-Purpose

A module designed for every conceivable use case, including ones that may never arise.

```python
# Too general: anticipates every possible need
class UniversalMessageDispatcher:
    def dispatch(self, channel, template, recipients, variables,
                 priority, scheduling, retry_policy, attachments,
                 tracking_config, ab_test_config, localization_config,
                 rate_limiting_config, webhook_callbacks):
        ...
```

The interface is so general that using it requires understanding 13 parameters. Most callers will use only a fraction of them.

### Somewhat General-Purpose (The Sweet Spot)

```python
# Somewhat general: covers current needs with a simple interface
class EmailService:
    def send(self, to: str, subject: str, body: str,
             attachments: list = None):
        ...
```

This covers welcome emails, password resets, invoices, and any future email type with a single, simple interface. It is general enough to handle all current use cases without special-case methods, but it does not try to handle SMS, push notifications, or A/B testing.

## The Key Question

> **"What is the simplest interface that will cover all my current needs?"**

This question is the practical tool for finding the sweet spot. It has three important parts:

1. **Simplest interface:** Minimize the number of methods, parameters, and concepts
2. **All current needs:** Do not design for hypothetical future requirements
3. **Cover:** The interface must actually work for every current use case without workarounds

### Applying the Question

**Step 1:** List all current use cases for the module.

**Step 2:** For each use case, identify what the caller needs from the module.

**Step 3:** Find the minimal set of methods and parameters that satisfies all callers.

**Step 4:** Check that no use case requires awkward workarounds.

**Example:**

A text editor needs to support:
- Inserting text at a position
- Deleting a range of text
- Replacing a range of text

Special-purpose approach:
```python
class TextEditor:
    def insert_text(self, position, text): ...
    def delete_range(self, start, end): ...
    def replace_range(self, start, end, new_text): ...
    def insert_heading(self, position, level, text): ...
    def insert_bullet_point(self, position, text): ...
    def delete_word(self, position): ...
    def replace_word(self, position, new_word): ...
```

Somewhat general-purpose approach:
```python
class TextEditor:
    def insert(self, position, text): ...
    def delete(self, start, end): ...
```

The general-purpose approach covers all cases with two methods. `replace` is just `delete` followed by `insert`. Headings and bullet points are just text with formatting characters. The interface is simpler and covers all current needs.

## Push Complexity Downward

**Principle:** It is more important for a module to have a simple interface than a simple implementation.

When complexity must exist somewhere in the system, it is better to put it inside a module (deepening it) than in the module's interface (burdening all callers).

### Why Downward, Not Upward?

| Complexity Location | Who Bears the Cost | Multiplier Effect |
|--------------------|-------------------|-------------------|
| Inside the module | The module's developer, once | 1x |
| In the interface | Every caller, every time they use it | Nx (where N = number of callers) |

A module with a complex implementation but simple interface imposes complexity on one developer (the module author). A module with a simple implementation but complex interface imposes complexity on every developer who uses it.

### Example: Connection Pooling

**Complexity pushed up (to callers):**
```python
# Every caller must manage pool lifecycle
pool = ConnectionPool(host, port, min_size=5, max_size=20)
conn = pool.acquire(timeout=5)
try:
    result = conn.execute(query)
finally:
    pool.release(conn)
# Must also handle pool exhaustion, stale connections, reconnection...
```

**Complexity pushed down (into the module):**
```python
# Caller just makes queries; pooling is internal
db = Database(connection_string)
result = db.query(sql, params)
# Pool management, connection lifecycle, retries all handled internally
```

### Example: Error Handling

**Complexity pushed up:**
```python
result = parser.parse(input)
if result.has_syntax_error:
    handle_syntax_error(result.syntax_error)
elif result.has_semantic_error:
    handle_semantic_error(result.semantic_error)
elif result.has_ambiguity:
    handle_ambiguity(result.ambiguity)
else:
    process(result.value)
```

**Complexity pushed down:**
```python
try:
    result = parser.parse(input)
    process(result)
except ParseError as e:
    # Module classifies and wraps all error types with clear messages
    show_error(e.message, e.location)
```

## Configuration Parameters: Complexity Amplifiers

Configuration parameters are one of the most common ways modules push complexity upward to callers. Each parameter represents a decision the module is refusing to make.

### The Problem

```python
# 11 decisions pushed to the caller
cache = Cache(
    max_size=1000,
    eviction_policy="lru",
    ttl_seconds=3600,
    cleanup_interval=300,
    max_memory_mb=256,
    serializer="json",
    compression=True,
    compression_level=6,
    stats_enabled=True,
    stats_interval=60,
    thread_safe=True,
)
```

Every parameter is a question the caller must answer. Most callers don't know the right answer and will either copy values from examples or guess. Wrong values cause subtle performance problems or bugs that are hard to diagnose.

### Better Approaches

| Strategy | How It Helps | Example |
|----------|-------------|---------|
| **Sensible defaults** | Module makes the decision unless overridden | `Cache()` works with reasonable defaults; override only what you need |
| **Auto-detection** | Module determines the right value at runtime | Auto-size based on available memory; auto-select compression based on data characteristics |
| **Progressive disclosure** | Simple API for simple use; options for advanced use | `Cache()` for basic use; `Cache.builder().with_eviction(lru).build()` for custom |
| **Convention over configuration** | Follow well-known patterns | Database connection reads from `DATABASE_URL` environment variable; no parameter needed |
| **Elimination** | Remove the parameter entirely | Instead of `thread_safe` parameter, always be thread-safe (the cost is usually negligible) |

### When Configuration Is Justified

Configuration parameters are justified when:
1. **Different callers genuinely need different values** (not just "might someday need")
2. **The module cannot determine the right value** (it lacks the information)
3. **The wrong default would cause real harm** (not just suboptimal performance)
4. **The decision changes between deployments** (environment-specific settings)

### The Test

For each configuration parameter, ask:
- "Can the module figure this out on its own?" If yes, remove the parameter.
- "Do most callers use the same value?" If yes, make it the default.
- "Will the caller know the right value?" If no, the parameter is shifting complexity, not simplifying.

## When Specialization Is Justified

Despite the general preference for general-purpose design, specialization is appropriate in certain situations:

### 1. Domain-Specific Modules

When a module embodies domain-specific knowledge that does not generalize.

```python
# Justified specialization: tax rules are inherently domain-specific
class USTaxCalculator:
    def calculate_federal_tax(self, income, filing_status, deductions):
        ...
    def calculate_state_tax(self, income, state):
        ...
```

A "general-purpose tax calculator" would need to know about every country's tax system. Specialization to US taxes hides substantial domain complexity behind a focused interface.

### 2. Performance-Critical Paths

When general-purpose abstractions introduce unacceptable overhead.

```python
# General-purpose: flexible but slow for the hot path
def transform(data, transformer_pipeline):
    for transformer in transformer_pipeline:
        data = transformer.apply(data)
    return data

# Specialized: optimized for the specific hot path
def transform_pixel_rgb_to_hsv(pixels: np.ndarray) -> np.ndarray:
    # SIMD-optimized, no dynamic dispatch, no allocation
    ...
```

### 3. User-Facing Interfaces

When the interface is used by end users (not developers), specialized vocabulary improves usability.

```python
# General-purpose API: flexible but requires domain knowledge
scheduler.create_recurring_task(
    interval=timedelta(days=7),
    start=next_monday(),
    handler=send_report
)

# Specialized API: matches user mental model
scheduler.send_weekly_report(day="monday", time="09:00")
```

### 4. Adapters and Bridges

When connecting two systems with incompatible interfaces, the adapter is inherently specific to both.

```python
class StripeToInternalPaymentAdapter:
    def convert_stripe_event(self, stripe_event) -> InternalPaymentEvent:
        ...
```

## Practical Guidelines

### When Designing a New Module

1. List all current use cases
2. Ask: "What is the simplest interface that covers all of these?"
3. Resist adding methods for hypothetical future use cases
4. Push complexity into the implementation, away from the interface
5. Default to slightly more general than you think you need -- it is usually simpler

### When Reviewing an Existing Module

| Signal | Problem | Action |
|--------|---------|--------|
| Many methods that differ only in parameters | Over-specialization | Merge into fewer, more general methods |
| Methods named after specific callers | Coupling to use cases | Rename around the concept, not the caller |
| Long parameter lists | Complexity pushed upward | Add defaults, auto-detect, or absorb decisions |
| Multiple modules with similar functionality | Opportunity for generalization | Extract a shared general-purpose module |
| Configuration that "nobody touches" | Parameters that should be defaults | Make them defaults or remove them |

### When Adding a Feature

Before adding a new method or parameter:
1. Can an existing method handle this with its current interface?
2. Can a slight generalization of an existing method handle this?
3. Does the new method introduce a special case that could be avoided?

The best features are those that require no interface changes because the existing abstraction already supports them.

## The Relationship to Information Hiding

General-purpose interfaces hide **use-case-specific knowledge**. When an interface is general, callers don't need to know about other callers' use cases. This is a form of information hiding that reduces dependencies between callers.

A special-purpose interface like `sendWelcomeEmail()` creates a dependency: every developer who sees it must understand the welcome email use case. A general-purpose interface like `send(to, subject, body)` hides all specific use cases, reducing the information each developer must hold in mind.

## Summary

The goal is not the most general-purpose design possible. It is the **simplest interface that covers all current needs**. This sweet spot produces modules that are:
- Simple to use (few methods, few parameters)
- Flexible enough for current needs (no workarounds required)
- Future-friendly (new use cases often fit the existing abstraction)
- Deep (general interfaces tend to hide more implementation complexity)

When in doubt, lean slightly toward more general -- it is usually simpler. But stop well before building a framework for every conceivable future need.

# Information Hiding and Information Leakage

Information hiding is the most important technique for achieving deep modules. It was first articulated by David Parnas in 1971 and remains the foundation of good software design. Information leakage is its opposite -- and one of the most common sources of unnecessary complexity.


## Table of Contents
1. [The Information Hiding Principle](#the-information-hiding-principle)
2. [Information Leakage](#information-leakage)
3. [Reducing Information Leakage](#reducing-information-leakage)
4. [Case Study: HTTP Request Handling](#case-study-http-request-handling)
5. [Information Hiding Checklist](#information-hiding-checklist)
6. [Relationship to Other Principles](#relationship-to-other-principles)

---

## The Information Hiding Principle

**Each module should encapsulate a few design decisions, and its interface should reveal as little as possible about those decisions.**

The "information" being hidden includes:
- Data representations and storage formats
- Algorithms and implementation strategies
- Communication protocols and wire formats
- Caching strategies and performance optimizations
- Error handling details and recovery mechanisms
- Hardware and OS-specific details
- Concurrency and synchronization strategies
- Configuration and default values

### Why Information Hiding Reduces Complexity

1. **Reduces dependencies:** If callers don't know about an implementation detail, they can't depend on it. Changes to hidden information affect only the module that owns it.

2. **Reduces cognitive load:** Developers using the module need to understand only its interface, not its internals. The hidden information is complexity that is removed from their mental model.

3. **Eliminates unknown unknowns:** When information is properly hidden, there is nothing hidden that callers need to know. The interface is the complete contract.

4. **Enables independent evolution:** Hidden implementations can be changed, optimized, or replaced without affecting any caller.

## Information Leakage

**Information leakage occurs when a design decision is reflected in multiple modules.** It creates a dependency on that decision: if it changes, all modules that know about it must change too.

### Forms of Information Leakage

#### 1. Interface Leakage (Most Obvious)

The module's interface directly exposes implementation details.

```python
# Leaking: interface exposes file format details
class UserStore:
    def save_as_json(self, user, filepath):
        ...
    def load_from_json(self, filepath) -> User:
        ...

# Hiding: interface abstracts storage format
class UserStore:
    def save(self, user):
        ...
    def load(self, user_id) -> User:
        ...
```

In the leaking version, every caller knows the storage format is JSON. Switching to a database requires changing every caller. In the hiding version, the storage mechanism is an internal decision.

#### 2. Back-Door Leakage (Most Subtle)

Two modules share knowledge that is not part of either interface, often through shared data formats, file conventions, or implicit protocols.

```python
# Module A writes:
with open("data.csv") as f:
    f.write(f"{user.id},{user.name},{user.email}\n")

# Module B reads (far away in the codebase):
with open("data.csv") as f:
    for line in f:
        id, name, email = line.strip().split(",")
```

Both modules know the CSV format (comma-separated, field order: id, name, email). This knowledge is not in either module's interface. If the format changes, both must change, but there is no compiler error or type check to guide you. This is a classic unknown unknown.

**Fix:** Create a single module that owns the data format:

```python
class UserCsvStore:
    def write(self, user):
        ...
    def read_all(self) -> list[User]:
        ...
```

#### 3. Temporal Leakage

Code is split based on when things happen rather than what knowledge they share.

```python
# Temporal decomposition: split by time
class HttpRequestReader:
    def read_headers(self, socket) -> dict:
        # Knows HTTP header format
        ...

class HttpRequestParser:
    def parse_body(self, headers, socket) -> Body:
        # Also knows HTTP header format (Content-Length, Content-Type)
        ...

class HttpResponseWriter:
    def write_response(self, socket, status, headers, body):
        # Also knows HTTP format
        ...
```

All three modules know the HTTP format, even though they are split into "read," "parse," and "write" phases. The temporal decomposition forces shared knowledge across module boundaries.

**Fix:** Organize by knowledge, not by time:

```python
class HttpConnection:
    def receive_request(self, socket) -> HttpRequest:
        # All HTTP format knowledge lives here
        ...
    def send_response(self, socket, response: HttpResponse):
        # All HTTP format knowledge lives here
        ...
```

#### 4. Decorator Leakage

The Decorator pattern is a frequent source of leakage because the decorator must understand the full interface of the object it wraps.

```java
// The decorator knows everything about InputStream's interface
class LoggingInputStream extends InputStream {
    private InputStream wrapped;

    public int read() {
        log("reading one byte");
        return wrapped.read();  // Pass-through
    }

    public int read(byte[] b) {
        log("reading into buffer");
        return wrapped.read(b);  // Pass-through
    }

    public int read(byte[] b, int off, int len) {
        log("reading with offset");
        return wrapped.read(b, off, len);  // Pass-through
    }

    // Must implement every InputStream method...
}
```

The decorator is shallow: it adds minimal functionality (logging) but must duplicate the entire interface. Every change to `InputStream` propagates to every decorator.

**Better alternatives:**
- Add logging inside the original class (flag-controlled)
- Use aspect-oriented approaches that don't require interface duplication
- Add a hook/callback mechanism inside the deep module

### How to Detect Information Leakage

| Signal | What It Means |
|--------|--------------|
| Two modules that "always change together" | They share knowledge that should be in one place |
| A data format or protocol mentioned in multiple files | Format knowledge has leaked |
| Tests that break when internal implementation changes | Test code has leaked knowledge about internals |
| Comments like "must match format in module X" | Explicit acknowledgment of leakage |
| Global constants shared across modules | Shared knowledge that may indicate coupling |
| Similar parsing/formatting code in multiple modules | Format knowledge is duplicated |

## Reducing Information Leakage

### Strategy 1: Merge Modules That Share Knowledge

If two modules share knowledge about a design decision, consider merging them. The result is one module that encapsulates the decision, with a single interface for the rest of the system.

**Before:**
```python
class ConfigReader:
    def read(self, path) -> dict:
        # Knows config file format
        ...

class ConfigApplier:
    def apply(self, config: dict):
        # Also knows config structure
        ...
```

**After:**
```python
class ConfigManager:
    def load_and_apply(self, path):
        # All config knowledge in one place
        ...
```

### Strategy 2: Create a New Module for Shared Knowledge

If merging is not practical (the modules are genuinely different concerns), extract the shared knowledge into a new module that both depend on.

**Before:**
```python
# In api_handler.py:
def format_error(code, message):
    return {"error": {"code": code, "message": message, "timestamp": now()}}

# In webhook_handler.py:
def format_error(code, message):
    return {"error": {"code": code, "message": message, "timestamp": now()}}
```

**After:**
```python
# In error_format.py:
def format_error(code, message):
    return {"error": {"code": code, "message": message, "timestamp": now()}}

# Both api_handler and webhook_handler import from error_format
```

### Strategy 3: Push Knowledge Downward

Move knowledge from callers into the module they call. This deepens the module and simplifies its interface.

**Before:**
```python
# Caller must know about retry strategy
for attempt in range(3):
    try:
        result = api_client.call(endpoint, data)
        break
    except TransientError:
        time.sleep(2 ** attempt)
```

**After:**
```python
# Module handles retries internally
result = api_client.call(endpoint, data)
# Retries, backoff, and error classification are hidden inside api_client
```

### Strategy 4: Separate Interface from Implementation Physically

Use language mechanisms to enforce information hiding:

| Language | Mechanism | Effect |
|----------|----------|--------|
| Python | Underscore prefix (`_private_method`) | Convention-based hiding |
| Java/C# | `private`/`protected` keywords | Compiler-enforced hiding |
| Go | Lowercase names (unexported) | Package-level hiding |
| Rust | `pub` vs non-`pub` | Module-level hiding |
| TypeScript | `private`, `#field`, module scope | Multiple levels of hiding |

### Strategy 5: Design Interfaces Around Abstractions

An interface should describe **what** the module does at an abstract level, not **how** it does it.

```python
# Leaking (how):
class Cache:
    def get_from_lru(self, key): ...
    def put_with_ttl(self, key, value, ttl_seconds): ...
    def evict_lru_entries(self, count): ...

# Hiding (what):
class Cache:
    def get(self, key): ...
    def put(self, key, value): ...
    # LRU policy, TTL, eviction are internal decisions
```

## Case Study: HTTP Request Handling

A web server must read an HTTP request (headers and body), route it to a handler, process it, and send a response. Here is how temporal decomposition causes leakage versus how information-based decomposition avoids it.

### Temporal Decomposition (Problematic)

```
Phase 1: Read raw bytes from socket → knows HTTP header format
Phase 2: Parse headers → knows HTTP header format
Phase 3: Read body based on Content-Length → knows header meaning
Phase 4: Route to handler → knows URL format from headers
Phase 5: Build response → knows HTTP response format
Phase 6: Write response to socket → knows HTTP format
```

HTTP format knowledge is spread across 6 phases. Changing anything about the HTTP handling requires touching all of them.

### Information-Based Decomposition (Better)

```
HttpProtocol module:
  - Owns ALL knowledge of HTTP format (headers, body, status codes)
  - Reads socket → produces HttpRequest objects
  - Takes HttpResponse objects → writes to socket

Router module:
  - Owns URL pattern matching
  - Maps HttpRequest to handler function

Handler modules:
  - Work with high-level HttpRequest/HttpResponse objects
  - Know nothing about raw HTTP format
```

Now HTTP format knowledge lives in one place. The router knows only about URL patterns. Handlers know only about request/response objects. Each module hides its specific knowledge.

## Information Hiding Checklist

For each module in your system, ask:

| Question | Desired Answer |
|----------|---------------|
| What design decisions does this module hide? | At least one significant decision |
| Could the implementation be replaced without changing callers? | Yes |
| Does the interface mention implementation-specific concepts? | No |
| Do tests verify behavior or implementation? | Behavior |
| Are there other modules that share knowledge about the same implementation detail? | No |
| If this module's internal format changes, how many other modules must change? | Zero |

If any answer is unsatisfactory, information is leaking and the design should be reconsidered.

## Relationship to Other Principles

- **Deep modules** achieve depth primarily through information hiding -- the hidden information is what makes them deep
- **General-purpose interfaces** hide specific use cases, which is a form of information hiding
- **Comments** should describe the interface (what is visible) without revealing hidden implementation details
- **Strategic programming** is the mindset that makes developers willing to invest effort in proper information hiding rather than taking shortcuts that leak

# Strategic vs Tactical Programming

The distinction between strategic and tactical programming is not about specific techniques -- it is about mindset. It determines whether a codebase improves or degrades over time, and it is the single biggest factor in long-term software quality.

## Two Mindsets

### Tactical Programming

**Goal:** Get the current feature working as quickly as possible.

**Characteristics:**
- The primary metric is "does it work?"
- Design happens incidentally (or not at all)
- Shortcuts are acceptable because "we'll fix it later"
- Each change introduces a small amount of complexity
- The codebase gradually degrades over months and years

**The inner monologue:**
- "This is a little hacky but it works"
- "I'll clean this up in the next sprint"
- "It's just one extra parameter, no big deal"
- "We don't have time for a proper abstraction"
- "It's technical debt but we'll pay it down later"

### Strategic Programming

**Goal:** Produce a great design that also happens to work.

**Characteristics:**
- The primary metric is "does this make the system simpler?"
- Design is deliberate and happens before implementation
- Working code is necessary but not sufficient
- Each change is an investment opportunity -- leave the code better than you found it
- The codebase gradually improves over months and years

**The inner monologue:**
- "This works, but is there a simpler way to express this interface?"
- "Before I add this feature, let me improve the module structure"
- "This parameter feels wrong -- the module should decide this internally"
- "Let me write the interface comment first to clarify the abstraction"
- "This will take an extra hour now but save many hours later"

## The Tactical Tornado

Ousterhout's most vivid concept: the **tactical tornado** is a developer who produces features at extraordinary speed, leaving a trail of complexity in their wake.

### Profile of a Tactical Tornado

| Trait | Description |
|-------|-------------|
| **Speed** | Ships features faster than anyone else on the team |
| **Heroics** | Often praised by management for delivering quickly |
| **Trail of wreckage** | Every module they touch becomes harder for others to work with |
| **Special cases everywhere** | Adds boolean parameters, flags, and one-off workarounds |
| **No refactoring** | Never goes back to clean up; always moving to the next thing |
| **Knowledge hoarding** | Often the only one who can work on their code (because it's incomprehensible to others) |

### The Damage

A tactical tornado produces code that:
1. **Works today** (they are often technically skilled)
2. **Is hard to understand** (optimized for writing speed, not reading speed)
3. **Is hard to modify** (full of implicit assumptions and hidden dependencies)
4. **Slows the entire team** (others spend hours understanding and working around the tornado's code)
5. **Cannot be safely changed** (unknown unknowns abound)

### The Math

If a tactical tornado produces features at 2x speed but creates code that is 3x harder to maintain, the team loses productivity as soon as anyone else touches that code. Over a year, the tornado's output is a net negative because the maintenance cost exceeds the development speed gain.

### How to Handle Tactical Tornados

| Approach | Details |
|----------|---------|
| **Code reviews** | Require design quality, not just correctness; reject PRs that add unnecessary complexity |
| **Design discussions** | Require interface design before implementation |
| **Complexity budgets** | Set explicit limits on interface size and module coupling |
| **Team metrics** | Measure team velocity over time, not individual output |
| **Mentoring** | Help the tornado see long-term impact; often they genuinely don't realize the cost |

## The Investment Mindset

### The 10-20% Rule

Ousterhout recommends spending roughly 10-20% of development time on design improvement. This is not a separate "refactoring phase" -- it is part of every feature's development.

**What the investment looks like:**

| Investment | Time | Payoff |
|-----------|------|--------|
| Write interface comments before code | 15-30 minutes | Catches bad designs before implementation |
| Improve a module's interface while adding a feature | 1-2 hours | Simplifies the module for all future changes |
| Refactor a function that has become too complex | 30-60 minutes | Reduces cognitive load for the next developer |
| Add missing comments to code you had to study | 15 minutes | Saves the next developer hours of reverse-engineering |
| Rename variables and functions for clarity | 15 minutes | Reduces obscurity throughout the module |
| Eliminate a configuration parameter by auto-detecting | 1-2 hours | Removes complexity for every caller |

### Why 10-20% Is Enough

You don't need massive refactoring projects. Small, continuous improvements compound:

```
Month 1:  Codebase quality: ████████░░ (80%)
Month 3:  Codebase quality: █████████░ (85%)  -- steady small investments
Month 6:  Codebase quality: █████████░ (90%)  -- improvements compound
Month 12: Codebase quality: ██████████ (95%)  -- team is highly productive
```

Compare with tactical programming:

```
Month 1:  Codebase quality: ████████░░ (80%)
Month 3:  Codebase quality: ███████░░░ (75%)  -- small shortcuts accumulate
Month 6:  Codebase quality: ██████░░░░ (65%)  -- velocity drops noticeably
Month 12: Codebase quality: ████░░░░░░ (45%)  -- team spends more time fighting code than building features
```

### Investment Opportunities in Every PR

Every pull request is an opportunity to improve the system. Some investments:

| Opportunity | Example |
|------------|---------|
| **Improve naming** | Rename `process()` to `validateAndPersistOrder()` |
| **Simplify an interface** | Remove a parameter that can be auto-detected |
| **Add missing comments** | Document the interface of a function you had to study |
| **Merge shallow classes** | Combine `OrderValidator` and `OrderService` |
| **Extract hidden information** | Move format knowledge from three modules into one |
| **Remove dead code** | Delete unused methods, parameters, or configuration options |
| **Fix temporal decomposition** | Merge `readConfig()` and `applyConfig()` into `loadConfig()` |

### The Boy Scout Rule with Teeth

"Leave the code better than you found it" is a common principle, but Ousterhout gives it teeth: every change should include at least one design improvement. Not every change needs a major refactoring, but every change should make some small improvement to the system's design.

## How Startups Should Approach Design

### The Myth: "We'll Fix It Later"

Many startups believe that design quality is a luxury they will invest in once they have product-market fit. This is almost always wrong.

**Why:**
1. "Later" never comes -- there is always another urgent feature
2. Technical debt compounds -- the cost of fixing grows exponentially
3. Early design decisions become architectural constraints that are extremely expensive to change
4. As the team grows, bad abstractions slow everyone down (not just the original author)
5. Velocity problems from poor design often look like "we need more engineers" problems

### The Reality

Startups that invest in design from day one:
- Ship features faster after the first few months (clean code is faster to modify)
- Onboard new developers faster (clear abstractions reduce ramp-up time)
- Have fewer production incidents (fewer unknown unknowns)
- Can pivot more easily (well-abstracted code adapts to new requirements)

Startups that take tactical shortcuts from day one:
- Ship features fast for the first few weeks
- Gradually slow down as complexity accumulates
- Spend increasing time debugging, not building
- Eventually face a "rewrite or die" decision (and rewrites usually fail)

### The Startup Investment

The 10-20% investment is even more affordable for startups because:
- The codebase is small, so improvements have outsized impact
- Early design decisions propagate through all future code
- The team is small, so design discussions are fast
- There is no legacy code to work around

**Practical startup approach:**
1. Spend 10% of time on design improvement -- not 0%, not 50%
2. Write interface comments for all public APIs
3. Don't create shallow classes "because that's how enterprise code works"
4. Refactor aggressively while the codebase is small and the cost is low
5. Establish code review norms that include design quality

## Culture: Facebook vs Google

Ousterhout contrasts two engineering cultures to illustrate the strategic vs tactical distinction.

### Facebook's "Move Fast and Break Things" (Tactical Culture)

| Aspect | Details |
|--------|---------|
| **Motto** | "Move fast and break things" (later changed to "Move fast with stable infrastructure") |
| **Incentive** | Ship features quickly; promotions based on launch velocity |
| **Design investment** | Minimal; design happens incidentally during implementation |
| **Result** | Large codebase with significant complexity; Facebook eventually had to invest heavily in infrastructure to manage the mess |
| **Lesson** | Tactical culture produces speed early but creates compounding problems |

Note: Facebook later changed their motto because the approach became unsustainable at scale. The "break things" philosophy worked for a small team but created enormous costs as the organization grew.

### Google's Design Culture (Strategic Culture)

| Aspect | Details |
|--------|---------|
| **Emphasis** | Design quality, readability reviews, code health |
| **Incentive** | Readability reviewers, design documents for significant changes |
| **Design investment** | Substantial; design documents and reviews before implementation |
| **Result** | Engineers reported being more productive on complex systems; easier to understand and modify unfamiliar code |
| **Lesson** | Strategic culture costs more upfront but compounds into higher long-term productivity |

### The Lesson

Neither extreme is right for every organization. But the evidence suggests that investing in design produces better outcomes over any timeframe longer than a few weeks. The key is not to choose between speed and quality but to recognize that strategic design **is** the fastest path when measured over months, not days.

## When to Invest Strategically

### Always Invest When:

| Situation | Why |
|-----------|-----|
| **Designing a new module interface** | Interface decisions are the hardest to change later |
| **A module's complexity is growing** | Small interventions now prevent major rewrites later |
| **Onboarding new team members** | Clear abstractions and comments dramatically reduce ramp-up time |
| **Multiple teams will use the code** | Interface quality multiplies across consumers |
| **The code is on a critical path** | Complexity in critical paths causes production incidents |

### Accept Tactical Approach When:

| Situation | Why | Caveat |
|-----------|-----|--------|
| **True prototype/throwaway code** | Code that will genuinely be deleted | Be honest -- most "prototypes" ship to production |
| **Tight deadline with defined scope** | The tactical code will be immediately followed by a design pass | Actually schedule the design pass; put it on the calendar |
| **Exploring an unfamiliar domain** | You don't know enough to design well yet | Plan to redesign once you understand the domain |

The critical discipline: if you take a tactical shortcut, acknowledge the debt and plan to repay it. Don't pretend the shortcut has no cost.

## Practical Exercises

### Exercise 1: Interface Audit

Pick a module you work with frequently. For each public method:
1. Write the interface comment you wish existed
2. Compare it to the actual interface
3. Identify unnecessary parameters, missing defaults, and leaked implementation details
4. Propose a simpler interface that covers all current use cases

### Exercise 2: Complexity Budget

For your next feature:
1. Before starting, write down the current complexity of the affected modules (interface size, number of dependencies, known pain points)
2. After finishing, measure again
3. Goal: the feature adds functionality without increasing complexity, or even reduces it

### Exercise 3: Tactical Tornado Detection

Review the last 10 PRs on your team:
1. Which PRs added new parameters to existing interfaces?
2. Which PRs added special-case handling?
3. Which PRs included comments explaining design decisions?
4. Which PRs simplified existing code while adding new features?

PRs that score poorly on these questions may indicate tactical programming.

### Exercise 4: Design Review

For your next code review, add these questions:
1. Does this change make the system simpler or more complex?
2. Is the interface simpler than the implementation?
3. Is information properly hidden?
4. Are there interface comments that describe the abstraction?
5. Could any configuration parameters be eliminated?
6. Are there pass-through methods that should be merged?

## Summary

The strategic vs tactical distinction is ultimately about whether you view design as an investment or a cost. Tactical programmers see design as overhead that slows them down. Strategic programmers see design as an investment that speeds them up. The evidence -- from individual careers, team productivity, and company outcomes -- overwhelmingly favors the strategic approach. The 10-20% investment in design is the highest-return activity in software engineering.