Self-hosting

Open resolver index

The self-hosted SQLite and FTS5 reference index for canonical works, provenance, refresh history and deterministic exports.

Status and boundary

The resolver index is an open local baseline for replacing latency-sensitive live catalogue fan-out. It stores metadata, source records, offers, rights statements and merge evidence; it does not store book files. It is not yet production-primary: production search still uses the six live adapters while full-catalogue imports, freshness, tombstones and deployment storage are completed.

The reference implementation uses Node's built-in SQLite module and SQLite FTS5. It needs Node 22.13 or later and can run without Netlify, a proprietary database or a hosted search vendor.

Run locally

npm run resolver:index -- init --db data/resolver-index/libreleaf.sqlite
npm run resolver:index -- ingest   --db data/resolver-index/libreleaf.sqlite   --input fixtures/resolver-index/sample.ndjson   --source checked-in-fixture
npm run resolver:index -- search   --db data/resolver-index/libreleaf.sqlite   --query Frankenstein   --region GB

Import and audit model

A cursor-exhausting snapshot builder emits deterministic NDJSON and an explicit completeness report. The official Project Gutenberg weekly CSV importer adds catalogue metadata without treating Gutenberg's ebook issue date as print publication year or inventing current file offers. The DOAB OAI-PMH importer exhausts opaque resumption tokens, archives and checksums every raw page, retains DOI/ISBN/licence evidence and keeps the feed's CC0 metadata licence separate from each book's reuse terms.

  • works store canonical display metadata and retained ranking explanations.
  • source_records store original source IDs, URLs, language/country metadata and fetch freshness.
  • offers store access type, URL, format, source rights statement, jurisdiction and applicability.
  • merge_decisions store the algorithm version and evidence for every canonical association.
  • refresh_runs retain successful and failed imports; a failed refresh never deletes the last known record.

Cutover criteria

The index becomes production-primary only after scheduled source-specific importers, reproducible input checksums, visible freshness, reviewed absence/tombstone rules, deployment storage and live fallback are verified. An incomplete snapshot may refresh known records but cannot prove that missing records were removed upstream.

JSON and CSV exports cover every indexed table. Optional PostgreSQL or search-engine adapters must preserve the same open result contract and ranking reasons; the SQLite path remains the reproducible reference implementation.

References