Documentation That Developers Actually Read

We had a microservice that three engineers built over six months. When all three left within a quarter, the remaining team spent four weeks just figuring out how to deploy it. There was no README, no architecture doc, no runbook. The code was "self-documenting" — which is another way of saying undocumented. That month of lost productivity cost more than every hour those engineers would have spent writing docs combined.

Why Most Documentation Fails

The problem isn't that teams don't write docs. It's that they write the wrong docs, in the wrong places, at the wrong time.

The Four Types of Developer Documentation

Not all documentation serves the same purpose. The Divio documentation framework (which we've adapted) splits docs into four types based on what the reader needs:

Most teams only write reference docs (if anything). The biggest gap is usually explanation docs — the "why" behind decisions. Six months later, nobody remembers why you chose PostgreSQL over MongoDB, or why the auth service is separate from the user service.

README: Your Project's Front Door

A good README answers the five questions every new developer has in the first 10 minutes:

1. What is this? One paragraph, no jargon
2. How do I run it? Copy-paste commands that work on a fresh machine
3. How is it structured? Directory layout with one-line explanations
4. How do I contribute? Branch strategy, PR process, code style
5. Who do I ask for help? Team contacts, Slack channel, relevant links

README Template

# Project Name

One-line description of what this does and who it's for.

## Quick Start

```bash
git clone <repo>
cp .env.example .env
docker compose up -d
# Visit http://localhost:3000
```

## Architecture

```
src/
├── api/          # REST endpoints (Express routes)
├── services/     # Business logic (no HTTP awareness)
├── models/       # Database models (Prisma)
├── jobs/         # Background workers (Bull queues)
└── utils/        # Shared helpers
```

## Development

- **Run tests:** `npm test`
- **Run linter:** `npm run lint`
- **Database migrations:** `npx prisma migrate dev`
- **Environment variables:** See `.env.example` (all vars documented inline)

## Key Decisions

- [Why PostgreSQL over MongoDB](docs/adr/001-database-choice.md)
- [Why we don't use an ORM for reports](docs/adr/003-raw-sql-reports.md)

## Team

- **Slack:** #project-name
- **On-call:** See PagerDuty schedule
- **Tech lead:** @username

The Quick Start section is the most important part. If a new developer can't get the project running in under 15 minutes by following your README, you have a documentation bug. We test our READMEs by having new hires follow them on day one — every stumble becomes a fix.

Architecture Decision Records (ADRs)

ADRs are the single most underused and highest-value documentation practice. They capture the why behind technical decisions — the context, options considered, and reasoning. Without them, future developers (including your future self) will either blindly follow decisions they don't understand or reverse decisions that had good reasons.

ADR Format

# ADR-007: Use Event Sourcing for Order Service

**Status:** Accepted
**Date:** 2025-09-15
**Context:**
The order service needs an audit trail of every state change.
We also need to reconstruct order state at any point in time
for dispute resolution. Current CRUD approach loses history.

**Options Considered:**
1. Soft deletes + audit log table — simple but duplicates data
2. Event sourcing — stores events as source of truth
3. CDC (Change Data Capture) from PostgreSQL — less intrusive but adds infra complexity

**Decision:**
Event sourcing (#2). The audit trail IS the data model, so there's
no sync problem between state and history. Accepted the tradeoff of
more complex queries (solved with read projections).

**Consequences:**
- Read queries need projections (materialized views rebuilt from events)
- Team needs training on event sourcing patterns (1-2 sprint ramp-up)
- Testing is harder (must set up event sequences, not just DB state)
- BUT: complete audit trail, easy replay, temporal queries "for free"

We store ADRs as numbered Markdown files in docs/adr/ inside the repo. They're version-controlled alongside the code they describe. When a decision is superseded, we mark the old ADR as "Superseded by ADR-XXX" — never delete them. The history of why things changed is as valuable as the current state.

API Documentation That Works

API docs have a different problem than other docs: they go stale fast because the API changes with every sprint. The solution is generating docs from code, not maintaining them separately.

Approaches by Effectiveness

For our PHP projects, we use inline docblocks that generate OpenAPI specs at build time. For API design patterns, we follow a separate set of conventions — but the documentation strategy is always: generate from code, validate in CI.

What Good API Docs Include

• Authentication: How to get credentials, where to send them, what errors look like when auth fails
• Request/response examples: Real JSON, not abstract schemas. Show the happy path AND error responses
• Rate limits: What they are, what headers to check, what happens when you hit them
• Changelog: What changed, when, and whether it's breaking. Dated entries, not just a version number

Runbooks: When Things Go Wrong at 2 AM

Runbooks are the documentation your on-call engineer reads while production is down. They need to be specific, step-by-step, and tested. A runbook that says "investigate the database" is useless at 2 AM when alerts are firing.

Runbook Template

## Runbook: Database Connection Pool Exhaustion

**Symptom:** API returns 503 errors. Grafana shows connection pool at 100%.
**Severity:** SEV-1 (user-facing outage)
**Escalation:** If not resolved in 15 min, page the database team lead

### Step 1: Confirm the issue
```sql
SELECT count(*) FROM pg_stat_activity WHERE state = 'active';
-- If > 90, connection pool is exhausted
```

### Step 2: Identify the culprit
```sql
SELECT query, state, wait_event_type, age(now(), query_start)
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY query_start ASC
LIMIT 20;
```
Look for long-running queries (age > 30s) or many identical queries.

### Step 3: Kill long-running queries (if safe)
```sql
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'active' AND query_start < now() - interval '5 minutes';
```
⚠️ This kills queries, not connections. Clients will retry.

### Step 4: Prevent recurrence
- Check if a missing index caused slow queries (EXPLAIN ANALYZE)
- Check if a recent deploy introduced an N+1 query
- Check if connection pool max_size needs increasing (current: 20)

### Previous incidents
- 2025-08-12: Caused by missing index on orders.user_id (ADR-012)
- 2025-06-03: Caused by analytics query running during peak hours

We write runbooks after every incident post-mortem. If the incident required non-obvious debugging steps, those steps become a runbook. The "Previous incidents" section is critical — it gives the on-call engineer context about what's caused this before.

Keeping Documentation Current

Stale docs are worse than no docs. They waste time and erode trust ("don't bother checking the wiki, it's always wrong"). Here's how we keep ours fresh:

Docs-as-Code: The Core Principle

Documentation lives in the same repository as the code it describes. Changes to code and docs happen in the same PR. This is the single most effective strategy for keeping docs current.

Automation That Helps

The "Two Reads" Rule

If a question about your system is asked twice (in Slack, in standups, during onboarding), the answer should become documentation. The first time is a conversation. The second time is a documentation gap.

Documentation Ownership

Every document has an owner — usually the team that owns the code it describes. During quarterly reviews, each team spends 2-3 hours reviewing their docs for accuracy. It's not glamorous work, but it prevents the slow death of institutional knowledge. We track doc ownership in a simple table in our onboarding guide.

What NOT to Document

Over-documentation is real. These things don't need docs:

• Self-evident code. getUserById(id) doesn't need a comment saying "gets user by ID"
• Implementation details that change frequently. Document interfaces and contracts, not internals
• Meeting notes that aren't decisions. If a meeting produced a decision, write an ADR. If not, it's noise
• Anything that can be auto-generated. Don't hand-write what a tool can produce from source code

The goal is a minimal set of docs that covers maximum ground. Every additional document is maintenance debt. Write the least documentation that prevents the most confusion.

Frequently Asked Questions