Concepts

Data model & ingestion

A Unit lives in two stores at once — a filterable row in the SQL catalog and a vector in the vector index — and ingestion keeps them in lockstep.

The two-store split#

Every Unit is written to both stores. The SQL catalog holds the rich, filterable, BM25-able row (text, modality, Locator columns, ACL fields). The vector index holds the embedding plus a compact metadata blob (≤10 KiB, up to 10 indexed keys) used for filter pushdown at query time.

Store	Holds	Used for
SQL catalog (SQLite)	Full Unit row + FTS5 index	Lexical (BM25) search, filtering, hydration, ACL
Vector index	Dense vector + pushdown metadata	Dense (ANN) search with scope/ACL pushdown
Object storage	Raw source bytes	Original file storage (source_uri)
Key-value store	Audit chain head	Tamper-evident audit linkage

Indexed pushdown metadata#

Two keys are isolation-critical and always injected: scope and acl_owner. The default indexed allocation also includes plane, modality, doc_id, and t_bucket (a coarse hourly time bucket for temporal filters).

Ingesting content#

POST /v1/ingest accepts a batch of Units across every retrieval plane. It requires the write permission (Bob, read-only, receives 403). Each Unit’s text (or derived caption / OCR / ASR) is embedded with @cf/baai/bge-m3, inserted into the catalog + FTS5, and upserted into its plane. A Unit is rejected if its scope is outside your effective scopes, and the Unit is written to whichever scope you target via X-Aegis-Scope (defaulting to your active scope).

Tier plane gate

A tier may only ingest into the planes it can query. The Free tier is text + doc_visual only — a batch that carries a visual or audio Unit is rejected with 402 upgrade_required rather than silently stored as unsearchable. Upgrade to Pro for the visual + audio planes (Pricing). The document ceiling is enforced the same way (402 quota_exceeded).

Supported source formats

Point source_uri at the raw bytes in object storage and set mime. The ingestion pipeline derives embeddable text per plane:

Plane	Formats	Derivation
`text`	Plain text, Markdown, code, logs, transcripts	Passthrough (chunk + embed).
`visual`	Images (PNG/JPEG/WebP), video shots	Caption / vision model → text.
`audio`	Audio + video (mp3/wav/mp4)	ASR (whisper) → utterance text; video is demuxed via the ASR queue.
`doc_visual`	Scanned pages, PDFs, tables/figures	OCR / doc-page caption → text.

POST/v1/ingest

curl

curl -s -X POST "https://superchargedb.krisch1218.workers.dev/v1/ingest" \
  -H "Authorization: Bearer aegis_sk_alice" \
  -H "Content-Type: application/json" \
  -d '{
    "units": [
      {
        "text": "Q4 backend renewals are expected to close in the first half of the quarter.",
        "scope": "acme/alpha/backend",
        "doc_id": "doc-forecast",
        "acl_groups": ["eng-backend"]
      }
    ]
  }'

Per-Unit fields

Field	Type	Default	Notes
`text`	string	—	Text / caption / OCR / ASR to embed + FTS-index (derived for non-text planes).
`scope`	string	active scope	Must be within your effective scopes.
`doc_id`	string	generated	Groups Units under one source.
`modality`	string	`text`	Fine-grained Unit type.
`plane`	string	from modality	text \| visual \| audio \| doc_visual.
`source_uri`	string	generated	Object-storage key / origin pointer.
`mime`	string	—	Content type of the source (e.g. video/mp4).
`locator`	Locator	—	Sub-unit span: time_range \| bbox \| char_span \| line_range.
`acl_owner`	string	your principal	ABAC owner.
`acl_groups`	string[]	your groups	ABAC groups that may read the Unit.

200 OK

{ "ingested": 1, "results": [ { "ok": true, "unit_id": "0192f3a1-..." } ] }

Locators on ingest

You can attach a rich Locator (bbox, time_range, char_span, line_range) directly on an ingested Unit, and the full pipeline (e.g. video → audio → ASR segments) produces them automatically. GET /v1/units/:id returns a structured locator rebuilt from the stored page/bbox/time/line columns so a UI can deep-link to the exact span.