Document management — architecture & design

1. Goals & non-goals

Goals

• Single source of truth. One canonical version of every document. No “the wiki said X but the slide deck says Y.”
• Dual-format by construction. Every doc is simultaneously machine-readable (raw markdown + structured frontmatter) and human-readable (rendered as a beautiful, searchable site with images, diagrams, cross-links).
• Agent-first ergonomics. Agents writing/updating docs nightly is the common case, not a power-user feature. The pipeline assumes programmatic writes and treats human edits as occasional.
• Browse, navigate, search — three-tier.
  1. Browse — visual hierarchy, sidebar tree, tags, related-docs, image galleries.
  2. Navigate — fast jumping by ID, by topic, by date, by author (human or agent name), by linked entity.
  3. Search — full-text (BM25) + semantic (vector) + filtered (frontmatter facets). All three exposed to humans via the UI and to agents via a single REST endpoint.
• Cheap to operate. No paid SaaS that charges per-seat or per-doc. Storage cost scales with bytes, not headcount.

Non-goals

• Building our own WYSIWYG editor. Humans use VS Code, Obsidian, or GitHub’s web editor.
• Real-time collaborative editing (Notion/Google-Docs style). Our write pattern is “one author/agent at a time, commit, move on.”
• Replacing the research vault (obsid-sponic/). That stays as Sonia’s BD research scratch space. Outputs that need to be shared get promoted into this system.

2. The shape of the answer

3. Storage layer — markdown + frontmatter

Directory layout

docs/
├── README.md                      # entry point
├── _meta/                         # taxonomies, tag definitions, glossary
│   ├── tags.yml
│   └── glossary.md
├── charter/                       # mission, values, governance
├── partners/                      # one file per partner org
│   ├── most.md
│   └── bujna-warszawa.md
├── locations/                     # one file per location/space
├── people/                        # one file per person
├── runbooks/                      # operational recipes
├── brand-kit/                     # brand decisions, voice, imagery rules
├── devtasks/                      # active eng/design work
├── decisions/                     # ADRs
└── assets/                        # images, diagrams, PDFs
    └── <doc-slug>/                # each doc's images in its own folder

Frontmatter schema

Every .md file has YAML frontmatter — the structured metadata that powers filtering, faceting, and graph navigation.

---
id: doc_partners_most                  # stable, never changes
title: MOST — Mazovia Open Spaces Trust
slug: most                             # url-safe, may differ from id
type: partner                          # partner | location | person | runbook | adr | charter | devtask
status: active                         # draft | active | archived
tags: [warsaw, urban-gardens, ngo]
links:                                 # explicit, typed cross-references
  - { rel: location, target: doc_locations_warsaw }
  - { rel: contact,  target: doc_people_jane_kowalski }
created_at: 2026-04-12
updated_at: 2026-05-04
updated_by: agent:partner-research-bot # or human:rahul
review_cadence: quarterly              # how often this should be re-checked
summary: >
  One-paragraph machine-readable summary. Used as the embedding
  source for "topic-level" semantic search. Keep under 500 chars.
---

Why this matters

• id is stable across renames → links don’t break.
• links gives us a typed graph for free; agents can traverse “all partners in Warsaw” without parsing prose.
• summary doubles as the embedding source for coarse-grained retrieval.
• updated_by distinguishes human edits from agent edits — useful for trust scoring and conflict resolution.

Why markdown specifically

• Already both machine- and human-readable. No transformation cost.
• Diffable in git, reviewable in PRs.
• Renders to HTML cleanly. Renders to plain text trivially. Renders to embeddings with no preprocessing beyond chunking.
• Every LLM was trained on tons of it. Agents write it natively.

4. Human layer — browse, navigate, search

The human-facing site is a static MkDocs Material site built nightly (and on every push) and deployed to Cloudflare Pages.

Browse — visual hierarchy

• Left sidebar — collapsible tree mirroring docs/ directory structure. Generated from the directory tree by the awesome-pages plugin (no manual nav maintenance).
• Top nav — the seven top-level sections (charter, partners, locations, people, runbooks, brand-kit, decisions). Always one click away.
• Right sidebar — auto-generated table of contents for the current page (h2/h3 headings).
• Tag pages — every tag in any doc’s frontmatter becomes a page at /tags/<tag>/ listing all docs with that tag, with summaries.
• Type pages — landing pages for each doc type (e.g. /partners/ shows a card grid of all partners with their summary, status, location, last-updated date).

Navigate — graph structure

• Backlinks panel — at the bottom of every doc, “Pages that link here.” Built from the links frontmatter graph.
• Related-docs — sidebar widget showing docs with high tag overlap or shared linked entities.
• Stable URLs — /p/<id> redirects to the canonical slug-based URL, so even if a doc is renamed, old links keep working.
• Breadcrumbs — show directory path + parent type for orientation.
• Last-modified + author — every page header shows who/what last touched it and when. Lets humans spot stale or agent-overwritten content quickly.

Search — three modes, one box

The search box does three things in parallel, merging results:

1. Full-text (lunr/BM25) — runs entirely in the browser, instant. Indexes title, headings, body, and frontmatter tags.
2. Semantic (pgvector) — sends the query to /api/docs/search, returns top-N semantically similar chunks. Slower (~200–500ms) but catches concept matches.
3. Faceted filter — sidebar filters: type, tag, status, updated-since. Stackable on top of either search mode.

Reading mode

• Inline image gallery (lightbox on click). Images stored next to the doc in assets/<slug>/.
• Mermaid diagrams rendered in-place.
• “Copy as markdown” button — gives humans the raw source for pasting into chat with an LLM.
• “Open in editor” button — deep-links to the GitHub web editor for the underlying file.

5. Machine layer — API for agents

The agent-facing API lives in apps/control (already a Next.js project on in.sponicgardens.com). It reads from a Postgres database that mirrors docs/ after each pipeline run.

Schema

-- one row per doc; mirrors a single .md file
create table public.docs (
  id              text primary key,
  slug            text unique not null,
  type            text not null,
  title           text not null,
  status          text not null,
  summary         text,
  tags            text[] not null default '{}',
  body_markdown   text not null,
  body_html       text not null,
  frontmatter     jsonb not null,
  path            text not null,
  git_sha         text not null,
  updated_at      timestamptz not null,
  updated_by      text not null,
  word_count      int not null,
  fts             tsvector generated always as (
                    setweight(to_tsvector('english', title), 'A') ||
                    setweight(to_tsvector('english', coalesce(summary, '')), 'B') ||
                    setweight(to_tsvector('english', body_markdown), 'C')
                  ) stored
);

-- chunks for fine-grained semantic search
create table public.doc_chunks (
  id          bigserial primary key,
  doc_id      text not null references public.docs(id) on delete cascade,
  chunk_idx   int not null,
  heading     text,
  content     text not null,
  token_count int not null,
  embedding   vector(1536) not null,
  unique (doc_id, chunk_idx)
);

-- typed graph between docs
create table public.doc_links (
  source_id text not null references public.docs(id) on delete cascade,
  target_id text not null references public.docs(id) on delete cascade,
  rel       text not null,
  primary key (source_id, target_id, rel)
);

Agent API — one endpoint, many modes

POST /api/docs/search

{
  "q": "partners doing urban gardens in warsaw",
  "mode": "hybrid",          // fts | semantic | hybrid (default)
  "filters": {
    "type": ["partner", "location"],
    "tags": ["warsaw"],
    "status": "active",
    "updated_since": "2026-01-01"
  },
  "include": ["summary", "tags", "body"],
  "limit": 10
}

Returns:

{
  "results": [{
    "id": "doc_partners_most",
    "slug": "most",
    "type": "partner",
    "title": "MOST — Mazovia Open Spaces Trust",
    "score": 0.87,
    "match_reasons": ["fts:warsaw", "semantic:urban-gardens"],
    "summary": "...",
    "tags": ["warsaw", "urban-gardens", "ngo"],
    "snippet": "...highlighted excerpt..."
  }],
  "total": 3,
  "took_ms": 142
}

Companion endpoints:

• GET /api/docs/{id} — full doc with frontmatter, body, links.
• GET /api/docs/{id}/related?rel=mentions — graph traversal.
• GET /api/docs/by-tag/{tag} — fast tag lookup.
• POST /api/docs/{id}/note — agents can append timestamped notes to a doc’s “agent notes” section without rewriting the file.

6. Information processing — when does each step happen?

Three timing tiers: upfront (one-time setup), on every commit (CI), nightly (cron). Plus one on-demand mode for agents writing docs in real time.

6.1 Upfront (one-time, ~1–2 days of work)

• Create top-level docs/ directory with the structure in §3.
• Write mkdocs.yml with Material theme, plugins (awesome-pages, tags, git-revision-date, mermaid2, glightbox, search).
• Define frontmatter JSON schema and a validator script that runs in CI.
• Create the Postgres tables via a Supabase migration.
• Write the four pipeline scripts (parser, indexer, embedder, link-graph-builder).
• Stand up /api/docs/search and companion endpoints in apps/control.
• Add a Cloudflare Pages project for the docs site.
• Migrate existing canonical docs (charter, runbook, brand-kit sources).

6.2 On every commit to main (~1–3 min)

GitHub Actions docs-pipeline workflow:

1. Lint — validate every .md file has required frontmatter fields; check that id is unique; check that all links.target IDs exist.
2. Build site — mkdocs build. Outputs static HTML to site/.
3. Diff — compare git diff HEAD~1 HEAD -- docs/ to find changed files.
4. Index changed docs — for each changed file: parse frontmatter + body, upsert into public.docs, re-chunk, re-embed via text-embedding-3-small, rebuild doc_links rows.
5. Deploy — push site/ to Cloudflare Pages.

This is the fast path. Agents committing one file get their changes searchable in ~90 seconds.

6.3 Nightly (~3 AM, ~10–30 min)

GitHub Actions docs-nightly workflow:

1. Full reindex audit — recompute embeddings + chunks for any doc whose git_sha in DB doesn’t match the current commit. Catches drift.
2. Stale detection — flag docs whose updated_at + review_cadence < now(). Surfaces a “Needs review” banner on the site.
3. Broken-link sweep — crawl the rendered site for 404s in internal links and image references; open a GitHub issue listing any found.
4. Agent-driven updates — invoke registered “doc-updater” agents (e.g. partner-research-bot, runbook-syncer). Each reads its assigned docs, decides if updates are needed, and commits changes.
5. Search-quality eval — replay canned queries against the search API; alert if recall@5 drops below threshold.
6. Backup — dump docs.* tables to R2 alongside the existing nightly DB backup.

6.4 On-demand (agent writing in real time)

• Read — hit /api/docs/search or /api/docs/{id}. Always reads the latest committed state.
• Append a note — POST /api/docs/{id}/note writes to a dedicated ## Agent notes section at the bottom of the file, commits, and triggers the on-commit pipeline.
• Propose a rewrite — agent opens a PR via gh rather than pushing to main. Humans review and merge.

6.5 The four pipeline scripts

apps/control/scripts/docs-pipeline/
├── parse.ts          # md + frontmatter → typed Doc object
├── index.ts          # Doc → upsert into public.docs
├── embed.ts          # Doc → chunks → embeddings → doc_chunks
└── link-graph.ts     # Doc → doc_links rows

Each script is callable independently (--doc <path>) and collectively from a --all orchestrator.

7. Design decisions — explicit tradeoffs

8. What we're NOT building yet

• Live collaborative editing. Add only if humans complain about merge conflicts.
• Branching/draft workflows beyond git PRs. The PR flow is enough.
• Permissions / access control on individual docs. Everything is internal; add RLS when needed.
• A custom in-browser editor. GitHub’s web editor + VS Code + Obsidian cover all needs.
• Multilingual rendering. MkDocs Material supports it; turn on when needed.
• Agent-to-agent doc subscriptions. Solve when first agent asks for it; schema supports updated_at > X polling.

9. Open questions for review

1. Site location: docs.sponicgardens.com (public) or in.sponicgardens.com/docs (gated intranet)? Default proposal: intranet, since most content is internal.
2. Embedding model: text-embedding-3-small (cheap, good) vs text-embedding-3-large (better, 6× cost)? Default proposal: small, with a documented upgrade path.
3. obsid-sponic/ promotion: Automatic or manual? Default proposal: manual, so Sonia can keep her scratch space scratchy.
4. Image handling: Embed via ![](r2-url) in markdown, or add an images: array to frontmatter? Default proposal: frontmatter array, so machines can find images per doc without parsing prose.

10. Implementation order

Appendix A — Static site generator deep comparison

The choice between MkDocs Material, Docusaurus, and Astro Starlight is the single biggest UX decision. All three render markdown to a fast, searchable site; they differ in toolchain, machine-friendliness, and feature ceiling.

A.1 At a glance

A.2 Machine-friendliness

This is the dimension that matters most for us, since Claude/agents will be the primary writers.

MkDocs Material

• Authoring: Pure markdown + YAML frontmatter. Zero MDX, zero JSX. Claude writes this in its sleep.
• Build: Single CLI — mkdocs build. Exit code 0/1.
• Inspection: HTML output is parseable; directory tree is trivially walkable by agents.
• Editing the theme: Partial overrides via Jinja2 templates + extra.css. Small surface area.
• Programmatic injection: Via mkdocs-gen-files plugin — Python hook generates markdown at build time.

Bottom line: very high. Python toolchain is the only friction.

Docusaurus

• Authoring: MDX by default. MDX is a footgun for agents — invalid JSX (a stray <, a curly brace) breaks the build with cryptic errors.
• Build: npm run build. Slow at 25s+ for our scale.
• Inspection: Generates sitemap.xml, sidebar JSON, search index. Plenty of structured artifacts.
• Editing the theme: React components. Every tweak touches a real component tree.
• Programmatic injection: Via plugin lifecycle hooks. More powerful but more code.

Bottom line: medium. MDX-by-default is the wart for agent-driven authoring.

Astro Starlight

• Authoring: Markdown + MDX, but Zod-validated frontmatter — invalid frontmatter produces clean, structured build errors pointing at the offending field.
• Build: astro build. Fast. Clean output.
• Inspection: Content collections API — agents can getCollection('docs') and get every doc with typed frontmatter as JS objects. Best inspection story.
• Editing the theme: Astro components are close to HTML+CSS. Simple for Claude.
• Programmatic injection: Via content collection loaders that pull from Postgres at build time. Most idiomatic.

Bottom line: very high — if we accept fewer existing plugins.

A.3 Cost breakdown (3-year TCO)

Assumed scale: ~2,000 docs, ~5 GB of images, internal-only audience (<100 readers/day).

All three are free to run. The difference is engineer-time TCO, where MkDocs Material wins by being boring.

A.4 Weighted decision matrix

A.5 Recommendation

MkDocs Material narrowly wins on the weighted matrix and decisively wins on time-to-value. The two hits against it — Python toolchain and weaker frontmatter validation — are both manageable.

Astro Starlight is the close second and the right pick if we ever want a fully custom theme. The unified TS toolchain + Zod-validated frontmatter is genuinely lovely for agent-driven work. Migration to Starlight is a 1–2 day job because all content is portable markdown.

Docusaurus is the wrong pick for us specifically because of MDX defaults. MDX is a great format for documentation that includes React components; for documentation written by agents, it’s a liability.

Appendix B — Why not Obsidian as the canonical store?

We already use Obsidian for obsid-sponic/ (Sonia’s BD research scratch space). Obsidian is excellent for one-human-thinking-out-loud, but as the canonical doc system it has structural problems:

• No server. Programmatic writes from agents require filesystem access (single-machine bottleneck) or community plugins of varying quality.
• No native API. Search-from-an-agent means parsing the vault yourself.
• Plugin fragility. The features we’d lean on (Dataview, templater, vector-search plugins) are maintained by hobbyists and occasionally break across Obsidian versions.
• Vault-as-database. Obsidian assumes one vault per workspace; multi-tenant or multi-app querying is awkward.