Skills DirectorySearch all skills ↗AI agent skill · by @google
Mantis Structural Index
Builds a content-addressed semantic-unit index from source code for structural context. Use when a pinned or live codebase is available and structural cross-reference data would improve research quality. Don't use for findings analysis, patching, or reporting.
Archived 2026-10-23 · Source updated 2026-09-13 · License: Apache-2.0. Check the original repository for the latest version.
SKILL.md instructions
# Structural Code Index Builder
This is an **optional first-class stage** in the Pass Lifecycle Contract. It
runs immediately after the snapshot is pinned (Block D), before the first
code-reading analysis stage (summarize/architecture). It only needs `CODE_ROOT`
\+ `SNAPSHOT_ID` and must not depend on architecture/KB.
## System Goal
Structural Code Index Builder. Builds a content-addressed semantic-unit index
from source code using capability-based per-partition backend selection,
degrading gracefully to grep. Provides `find_callers(symbol)`,
`get_function_boundary(file, line)`, and call-site awareness to improve LLM
reasoning quality during discovery — supplementing (never replacing) grep-based
call-site discovery with structural data. The index is composed from independent
semantic compilation units, persisted as a manifest + SQLite catalog, and
exposed through a bounded query helper.
## Command Definition
- **Command:** `/mantis-structural-index`
- **Description:** Build a content-addressed semantic-unit index from source
code under `CODE_ROOT`.
- **Arguments (optional; supplied by the orchestrator, consumed by Block A):**
`--snapshot_root`/`--snapshot_id`/`--state_root`. All absent → MODE-OFF/legacy
mode (reads source from the current directory, writes index to
`./workspace/kb/structural_index/`).
## Input/Output Contract
- **Reads**:
- `workspace/.mantis_state.json` (to read `active_snapshot` for provenance
checking and snapshot-aware rebuild logic).
- `workspace/kb/structural_index/manifest.json` (to check `snapshot_id` for
reuse-on-match idempotency — primary check).
- `workspace/kb/structural_index.jsonl` (backward-compat provenance check if
manifest absent).
- `workspace/kb/structural_index/units/` (content-addressed cache for
incremental unit reuse).
- `workspace/kb/structural_index/native/` and sidecar `provenance.json` files
(prebuilt index attachments and metadata manifests).
- CODE_ROOT source files (via generated helper script — the script parses all
source files under `CODE_ROOT`).
- **Writes**:
- `workspace/kb/structural_index/manifest.json` (STATE-RELATIVE — atomic
commit point, written LAST).
- `workspace/kb/structural_index/catalog.sqlite` (STATE-RELATIVE —
query-optimized serving store).
- `workspace/kb/structural_index/units/` (STATE-RELATIVE — content-addressed
immutable unit outputs).
- `workspace/kb/structural_index/shards/` (STATE-RELATIVE — partitioned
serving data for large corpora).
- `workspace/kb/structural_index/native/` (STATE-RELATIVE — prebuilt index
attachments: SCIP, Kythe, LSIF).
- `workspace/kb/structural_index/tmp/` (STATE-RELATIVE — temporary objects
during build).
- `workspace/kb/structural_index.jsonl` (STATE-RELATIVE — compatibility
pointer; full export below threshold).
- `workspace/helpers/build_structural_index.py` (STATE-RELATIVE — the builder
helper script).
- `workspace/helpers/query_structural_index.py` (STATE-RELATIVE — the query
helper script).
- **Preconditions**:
- Source files must exist under `CODE_ROOT`. If `CODE_ROOT` is not resolved
(MODE-OFF and no readable `active_snapshot`), build against the current
directory with `snapshot_id` set to `"unknown"`. Do NOT skip — this is the
standalone-efficiency case.
- **Inert until wired:** This skill returns an empty index until a caller
invokes it (the harness, `mantis-plan`, or `mantis-researcher`). It never
fails — it simply returns an empty index if tools are unavailable or source
cannot be parsed.
- **Idempotency Guarantee**:
- Read-only on `CODE_ROOT`. Writes only to STATE-RELATIVE paths. Re-running
with the same `CODE_ROOT` and `SNAPSHOT_ID` reuses the existing index
(manifest `snapshot_id` match) rather than rebuilding — except in MODE-OFF,
where it always rebuilds. Individual semantic units are reused across
snapshots when their content-addressed cache keys match (incremental
rebuild).
## Instructions
### Step 0: Locator Resolution (run first)
```
LOCATOR RESOLUTION (before reading ANY target code or artifact):
0. ROLE: If this skill NEVER reads target source (report, calibrate, reflect),
you are a FINDINGS-ONLY stage: skip steps 2-6; still read active_snapshot from
state for provenance/annotation; NEVER stop merely because a code root is unset.
1. Determine CODE_ROOT, in this priority order:
a. If --target_root is passed on THIS invocation, CODE_ROOT = --target_root.
It is AUTHORITATIVE and OVERRIDES SNAPSHOT_ROOT and the state fallback
(used when a caller hands you a prepared tree, e.g. a patched shadow).
b. Else if --snapshot_root (or SNAPSHOT_ROOT) is passed, use it.
c. Else read state_root/workspace/.mantis_state.json (state_root from
--state_root if passed, else ./workspace/... relative to the current dir)
-> active_snapshot.root / .snapshot_id / .snapshot_pinned.
d. Else (no arg AND no readable active_snapshot): CODE_ROOT = current directory,
treat snapshot_pinned = false (MODE-OFF). Do NOT stop.
2. SENTINEL CHECK (only if snapshot_pinned is true AND you did NOT take path 1a):
verify CODE_ROOT/.mantis_snapshot_id exists and equals SNAPSHOT_ID. If missing
or different -> STOP "snapshot sentinel mismatch". (A --target_root tree (1a) is
deliberately mutated and is sentinel-EXEMPT.)
3. PATH FIELDS:
- SNAPSHOT-RELATIVE (read under CODE_ROOT): code_paths entries; plan target_files
that are file paths. Strip ONLY a trailing ":<digits>". A code_paths entry
containing "://" is a URL/endpoint, NOT a file read. A code_paths entry that is
NOT of the form <existing-path>:<integer> is a non-source LOCATOR
(symbol/offset/endpoint): only check that the artifact/symbol exists; skip ALL
line-range and line-existence logic.
- STATE-RELATIVE (read/write under state_root/workspace, NEVER prefix CODE_ROOT):
kb_references, repro_file_path, reattack_file_path, helper scripts, report
files, and all state/findings JSON.
4. Never WRITE under CODE_ROOT when snapshot_pinned is true. Any command that
compiles, generates, or writes artifacts MUST run in a PRIVATE SHADOW copy
(mktemp -d from CODE_ROOT), never with cwd=CODE_ROOT. Read-only inspection may
cd into CODE_ROOT.
5. VCS-METADATA CARVE-OUT: history-log extraction and any VCS diff/blame command
run in the LIVE repository root (which still has .git/.hg/.repo), NOT CODE_ROOT
(the snapshot copy strips VCS metadata). Do NOT stop merely because CODE_ROOT
lacks .git/.hg/.repo.
6. Every shell command uses ABSOLUTE paths and sets its own working directory on
that call. Do NOT assume the working directory persists between calls.
```
This is a CODE-READING stage — it reads target source files under CODE_ROOT via
the helper script. Block A step 0's findings-only skip does NOT apply.
### Step 1: Idempotency / Freshness Check (MANDATORY FIRST)
**MANDATORY FIRST STEP:** Before writing any build scripts, probing backends, or
extracting symbols, check for MODE-OFF or an existing manifest:
1. **MODE-OFF check (FIRST):** In MODE-OFF (`SNAPSHOT_ID` is `"unknown"` or
absent) → **always rebuild**. Do NOT reuse a previous `"unknown"` index,
because the live tree is mutable and `"unknown"` is a constant (not a
freshness signal). Skip directly to Step 2 (individual units whose content
has not changed may still hit the content-addressed cache).
2. If `workspace/kb/structural_index/manifest.json` exists, read its
`snapshot_id` field.
3. If `snapshot_id` matches the current `SNAPSHOT_ID` (and `SNAPSHOT_ID` is NOT
`"unknown"`) → **reuse the index immediately and STOP**. Do NOT invoke Step
2, probe backends, or write build scripts. This bounds cost across
retries/crash-resume.
4. If `manifest.json` is absent but `workspace/kb/structural_index.jsonl` exists
(backward compat), read its provenance header (`_provenance`, `snapshot_id`
keys). If `snapshot_id` matches (and is NOT `"unknown"`) → **reuse and
STOP**. Otherwise proceed to rebuild.
5. If `SNAPSHOT_ID` differs → proceed to rebuild (Steps 2–5). For incremental
reuse: before rebuilding a semantic unit, check `units/` for a
content-addressed cache hit (see Content-Addressed Cache Key below). A cache
hit reuses the unit output without re-extraction.
### Step 2: Select Backend per Partition
Partition the codebase into semantic units (see Per-Language Semantic Units
below). For each unit, select the most precise backend available in this
environment using a **capability-based, per-partition decision** — not a single
global ladder. The following are **examples, not an exhaustive enum**. The
helper probes each tier per partition and selects the highest available:
1. **Snapshot-matched SCIP / LSIF / Kythe / clangd-static / remote index**
(precision: `semantic`) — if a pre-built index matching the current
`snapshot_id` or `root_fingerprint` is available. Most precise: full
type-aware cross-reference, call hierarchy, and hover/signature data.
2. **Compiler / typechecker-backed extractor** (precision: `typecheck`) — if
`compile_commands.json`, build context, or typechecker is available.
Type-accurate symbol resolution and call edges.
3. **Language-aware AST extraction** (precision: `ast`) — if `tree-sitter`,
`ast-grep`, or a language-specific parser is available. Full AST parsing:
function boundaries, call expressions, signatures.
4. **Symbol-only extraction** (precision: `symbol-only`) — if `ctags` or
equivalent is on `PATH`. Symbol table only (function definitions, locations —
no call graph). Call-site extraction uses a lightweight regex pass within
known function boundaries.
5. **Heuristic fallback** (precision: `heuristic`) — Python stdlib regex pass
over source files. Identifies function definitions and call patterns using
language-agnostic heuristics. Less precise but zero-dependency.
6. **Coverage-only manifest + lexical fallback** (precision: `coverage-only`) —
grep; no structural index is written. Manifest records `status: "empty"`.
Consumers fall back to grep-based discovery (today's behavior byte-for-byte).
**Native Index Probing & Resolution Rules**:
- **Probe instruction**: Before evaluating per-partition backends, probe
`workspace/kb/structural_index/native/` and subdirectories
`native/{scip,lsif,kythe}/` for prebuilt index files.
- **Snapshot-declaration convention**: Because native formats (SCIP, LSIF,
Kythe) do not embed snapshot identity directly in their binary payload,
prebuilt indexes MUST declare their target snapshot using a sidecar
`provenance.json` manifest located at
`workspace/kb/structural_index/native/provenance.json` or
`native/<kind>/provenance.json`. The manifest contains an array of
attachments:
`[{"kind": "scip|lsif|kythe", "path": "...", "snapshot_id": "...", "root_fingerprint": "...", "language": "...", "indexer": "...", "precision": "semantic", "files": [...]}]`.
A native index is matched if its declared `snapshot_id` equals `SNAPSHOT_ID`
(when `SNAPSHOT_ID != "unknown"`) or its `root_fingerprint` matches the
workspace's calculated root fingerprint. If the `files` array is absent or
empty, treat the native index as covering no individual files directly (record
in `manifest.native_indexes` but do not update `coverage` rows; fall through
to lower tiers for all files).
- **Record-and-Defer Ingestion Rule**: Parsing raw binary native indexes (e.g.
SCIP protobuf) in pure Python without dependencies is costly and complex.
Option A builder scripts MUST detect matching prebuilt native indexes, record
their entries in `manifest.native_indexes`, and set the `coverage` table
`backend` (e.g., `"scip"` or `"scip-clangd"`) for all files listed in the
provenance manifest. If `catalog.sqlite` is NOT populated with symbols from
the native index (raw binary deferred to harness/MCP readers), set
`coverage.status = "deferred"` and `precision = "deferred"` (with
`indexed_files = 0`). This prevents the query helper from claiming an
un-ingested partition is "authoritative empty" at `semantic` precision,
ensuring consumers run the mandatory grep fallback. When `catalog.sqlite` IS
populated (e.g., via Option B pre-ingestion or `scip-to-sqlite`), set
`precision = "semantic"` and `status = "indexed"`.
**LSP is NOT equivalent to SCIP / LSIF.** LSP is an interactive protocol whose
workspace state may be partial or mutable. Use it only when the server can
demonstrate snapshot identity AND complete workspace coverage. A running
language server does not automatically qualify as a `semantic` backend.
**SCIP merging.** SCIP explicitly permits merging complementary information from
indexers with different precision levels. Its format records indexer version and
per-document language metadata. The builder MAY merge results from multiple
indexers (e.g., a SCIP index for Go + tree-sitter for Python) within a single
catalog.
The determinism lives in a runtime-generated versioned helper
(`build_structural_index.py`, `# MANTIS_HELPER_VERSION = 5`, grep-and-regenerate
on reuse) that probes and selects backends per partition. No shipped binaries;
air-gapped-safe.
**Deterministic partial coverage.** On very large source trees, a full rebuild
can dominate stage-0 wall-clock. Replace discovery-order truncation with a
deterministic priority queue:
1. **Explicit target files and symbols** (from `plan.json`, if available).
2. **Changed units and known reverse dependencies** (from Block E diff +
reverse-dependency edges in the existing catalog).
3. **Containing packages/modules and direct imports** of tier 1+2 units.
4. **Remaining units in normalized path order**.
Apply deterministic `max_units` (default: 10000) or `max_source_bytes` (default:
500MB) bounds to that queue. Persist deferred units in the manifest so another
invocation can resume. Statuses: `complete`, `partial`, `empty`, `failed`.
A consumer should only interpret "no callers" strongly when the relevant
partition is complete, snapshot-matched, covered, and semantically precise.
Otherwise the answer is "no indexed callers", followed by the existing lexical
fallback. The index is HINT-only, so partial coverage is safe (grep remains
authoritative).
### Step 3: Write and Run Helper Scripts
Two runtime-generated helpers are used. Both follow the grep-and-regenerate
pattern: before reuse, grep the first line for the version marker; if absent or
a different integer, REGENERATE.
#### Builder: `build_structural_index.py`
1. Write the builder to `workspace/helpers/build_structural_index.py`. The FIRST
LINE MUST be exactly `# MANTIS_HELPER_VERSION = 5`. Before reusing an
existing helper, grep its first lines for `MANTIS_HELPER_VERSION = 5`; if
that marker is absent or a different integer, REGENERATE the helper.
2. The builder partitions the codebase into semantic units (see Per-Language
Semantic Units), computes content-addressed cache keys, checks `units/` for
reuse, selects a backend per partition (Step 2), extracts symbols + edges,
and writes results to `catalog.sqlite`.
3. The builder must extract:
- **Symbols**: `symbol_id`, `name`, `qualified_name`, `namespace`,
`language`, `file_path`, `start_line`, `end_line`, `kind`, `signature`,
`backend`, `precision`.
- **Call edges**: `caller_id`, `callee_id`, `callee_name`, `file_path`,
`line`, `edge_kind`.
- **Function boundaries**: `symbol_id`, `file_path`, `start_line`,
`end_line`, `signature`, `language`.
- **Coverage**: `file_path`, `indexed`, `backend`, `precision`,
`unit_cache_key`, `status`.
4. The builder writes the manifest LAST (Step 4) and the compatibility pointer.
5. The builder MUST use ABSOLUTE paths and set its own working directory (Block
A step 6). It MUST NOT write anything under `CODE_ROOT` when
`snapshot_pinned` is true (Block A step 4). Backends that produce sidecar
files (ctags `tags`, cscope `cscope.out`, clangd cache) MUST be redirected to
STATE-RELATIVE paths: `ctags -f <state>/helpers/tags`,
`cscope -f <state>/helpers/cscope.out`,
`CLANGD_INDEX_STORAGE=<state>/helpers/`. Read-only LSP/SCIP queries to a
running server need no redirect.
#### Query helper: `query_structural_index.py`
1. Write the query helper to `workspace/helpers/query_structural_index.py`. The
FIRST LINE MUST be exactly `# MANTIS_HELPER_VERSION = 5`. Before reusing an
existing helper, grep its first lines for `MANTIS_HELPER_VERSION = 5`; if
absent or a different integer, REGENERATE.
2. The query helper provides bounded, paginated operations against
`catalog.sqlite` (or a remote endpoint — identical API). It IS the
consumption contract (see Query Interface below).
3. If `catalog.sqlite` is absent but `structural_index.jsonl` exists, the query
helper falls back to linear scanning of the JSONL file (slower but
functional). If both are absent, it returns empty results with
`coverage.partition_status = "empty"`.
### Step 4: Write Manifest and Commit
1. Write the manifest to `workspace/kb/structural_index/tmp/manifest.json`
first. Then atomically rename it to
`workspace/kb/structural_index/manifest.json`. This is the atomic commit
point — the manifest is written LAST, after all units, catalog, and coverage
data are written.
2. Write a compatibility pointer to `workspace/kb/structural_index.jsonl`
(STATE-RELATIVE — NEVER under `CODE_ROOT`). Below a configurable threshold
(default: 10K records), emit a complete JSONL export (provenance header + all
records). Above the threshold, emit only a provenance header with
`compat_pointer.full_export = false` and `compat_pointer.symbol_count` set.
Large consumers MUST use the query interface.
3. The manifest `provider.backend_versions` field records which backend was used
per language (e.g.,
`{"go": {"backend_name": "scip-clangd", "precision": "semantic"}, "python": {"backend_name": "tree-sitter", "precision": "ast"}}`),
so consumers know the precision level.
4. Interrupted builds leave unreferenced temp objects in `tmp/` without
corrupting the last published index. On resume, check `manifest.json` status
and `snapshot_id` (Step 1).
### Step 5: Return Results / Notify Caller
1. Return the path to `manifest.json`, `catalog.sqlite`, and the query helper.
Include a summary (unit count, symbol count, call-edge count, backends used,
coverage status).
2. If the index is empty (no tools available or no source files found), notify
the caller: "Structural index is empty — structural context unavailable."
3. Do not notify the user directly — this skill is invoked as a sub-agent by the
harness, planner, or researcher.
## Index Schema and On-Disk Contract
### On-Disk Layout
```
workspace/kb/structural_index/
├── manifest.json # Atomic commit point — written LAST
├── catalog.sqlite # Query-optimized serving store (both directions indexed)
├── units/ # Content-addressed immutable unit outputs
│ └── ab/cd/abcdef... # sha256 prefix sharding (2+2 hex dirs)
├── shards/ # Partitioned serving data (large corpora)
│ └── shard_0000.sqlite
├── native/ # Prebuilt index attachments (SCIP, Kythe, LSIF)
│ ├── provenance.json # Prebuilt index provenance manifest
│ ├── scip/
│ └── kythe/
└── tmp/ # Temporary objects during build
workspace/helpers/
├── build_structural_index.py # Builder (MANTIS_HELPER_VERSION = 5)
└── query_structural_index.py # Query helper (MANTIS_HELPER_VERSION = 5)
workspace/kb/structural_index.jsonl # Compatibility pointer
```
### Manifest Schema (`manifest.json`)
```json
{
"schema_version": 1,
"snapshot_id": "<SNAPSHOT_ID or 'unknown'>",
"root_fingerprint": "<sha256 of sorted (path, content_sha256) for all source files>",
"status": "complete|partial|empty|failed",
"provider": {
"kind": "local-build|baseline+overlay|remote",
"catalog": "catalog.sqlite",
"backend_versions": {
"cpp": {"backend_name": "tree-sitter", "backend_version": "0.20.8", "precision": "ast"},
"go": {"backend_name": "scip-clangd", "backend_version": "0.2.3", "precision": "semantic"}
}
},
"units": {"total": 0, "reused": 0, "rebuild": 0, "failed": 0},
"coverage": {"total_files": 0, "indexed_files": 0, "failed_files": 0, "deferred_files": 0},
"shards": [{"id": "", "path": "", "checksum": "", "partition_key": "", "symbol_count": 0, "edge_count": 0}],
"deferred_units": [{"unit_id": "", "language": "", "files": [], "priority": 4, "reason": ""}],
"native_indexes": [{"kind": "scip", "path": "", "snapshot_id": "", "root_fingerprint": "", "language": "", "indexer": "", "precision": ""}],
"baseline": {"source": "ci|local|none", "snapshot_id": "", "manifest_path": ""},
"overlay": {"units_added": 0, "units_modified": 0, "files": []},
"compat_pointer": {"path": "structural_index.jsonl", "full_export": true, "symbol_count": 0},
"created_at": "<ISO 8601>",
"build_duration_ms": 0
}
```
**Atomic commit**: The manifest is written LAST (atomic rename from `tmp/`).
Interrupted builds leave unreferenced temp objects without corrupting the last
published index.
### SQLite Catalog Schema (`catalog.sqlite`)
```sql
CREATE TABLE IF NOT EXISTS schema_meta (
key TEXT PRIMARY KEY, value TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS symbols (
symbol_id TEXT PRIMARY KEY,
name TEXT NOT NULL,
qualified_name TEXT NOT NULL,
namespace TEXT,
language TEXT NOT NULL,
file_path TEXT NOT NULL,
start_line INTEGER NOT NULL,
end_line INTEGER,
kind TEXT NOT NULL,
signature TEXT,
backend TEXT NOT NULL,
precision TEXT NOT NULL CHECK (precision IN ('semantic','typecheck','ast','symbol-only','heuristic','deferred','coverage-only')),
corpus TEXT DEFAULT 'default',
partition_key TEXT,
unit_cache_key TEXT,
source_layer TEXT NOT NULL DEFAULT 'baseline'
);
CREATE TABLE IF NOT EXISTS call_edges (
edge_id INTEGER PRIMARY KEY AUTOINCREMENT,
caller_id TEXT NOT NULL,
callee_id TEXT,
callee_name TEXT NOT NULL,
file_path TEXT NOT NULL,
line INTEGER NOT NULL,
edge_kind TEXT NOT NULL CHECK (edge_kind IN ('direct','indirect','virtual','macro','unresolved')),
corpus TEXT DEFAULT 'default',
partition_key TEXT,
source_layer TEXT NOT NULL DEFAULT 'baseline',
FOREIGN KEY (caller_id) REFERENCES symbols(symbol_id),
FOREIGN KEY (callee_id) REFERENCES symbols(symbol_id)
);
CREATE TABLE IF NOT EXISTS function_boundaries (
symbol_id TEXT PRIMARY KEY,
file_path TEXT NOT NULL,
start_line INTEGER NOT NULL,
end_line INTEGER NOT NULL,
signature TEXT,
language TEXT NOT NULL,
FOREIGN KEY (symbol_id) REFERENCES symbols(symbol_id)
);
CREATE TABLE IF NOT EXISTS coverage (
file_path TEXT NOT NULL,
indexed INTEGER NOT NULL DEFAULT 0,
backend TEXT,
precision TEXT,
unit_cache_key TEXT,
status TEXT NOT NULL DEFAULT 'pending',
corpus TEXT DEFAULT 'default',
partition_key TEXT,
PRIMARY KEY (file_path, corpus)
);
CREATE TABLE IF NOT EXISTS unit_cache (
cache_key TEXT PRIMARY KEY,
unit_id TEXT NOT NULL,
language TEXT NOT NULL,
extractor_name TEXT NOT NULL,
extractor_version TEXT NOT NULL,
file_count INTEGER NOT NULL,
symbol_count INTEGER NOT NULL,
edge_count INTEGER NOT NULL,
source_bytes INTEGER NOT NULL,
created_at TEXT NOT NULL,
snapshot_id TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS deferred_units (
unit_id TEXT PRIMARY KEY,
language TEXT NOT NULL,
files TEXT NOT NULL,
priority INTEGER NOT NULL,
reason TEXT NOT NULL,
cache_key TEXT,
created_at TEXT NOT NULL
);
-- Indexes: both directions!
CREATE INDEX IF NOT EXISTS idx_symbols_name ON symbols(name);
CREATE INDEX IF NOT EXISTS idx_symbols_qualified ON symbols(qualified_name);
CREATE INDEX IF NOT EXISTS idx_symbols_file ON symbols(file_path, start_line);
CREATE INDEX IF NOT EXISTS idx_symbols_lang ON symbols(language);
CREATE INDEX IF NOT EXISTS idx_edges_caller ON call_edges(caller_id);
CREATE INDEX IF NOT EXISTS idx_edges_callee ON call_edges(callee_id);
CREATE INDEX IF NOT EXISTS idx_edges_callee_name ON call_edges(callee_name);
CREATE INDEX IF NOT EXISTS idx_edges_file ON call_edges(file_path, line);
CREATE INDEX IF NOT EXISTS idx_boundaries_file ON function_boundaries(file_path, start_line, end_line);
CREATE INDEX IF NOT EXISTS idx_coverage_status ON coverage(status);
CREATE INDEX IF NOT EXISTS idx_deferred_priority ON deferred_units(priority);
```
**Partitioning**: When symbols exceed 500K or edges exceed 2M, partition into
shard SQLite files under `shards/` by `lang:{language}:bucket:{NN}` (stable
hash). Each shard has the same schema. The query helper routes queries to the
correct shard(s) based on the manifest.
### Content-Addressed Cache Key
```python
cache_key = sha256(
schema_version +
extractor_name@version +
language +
compile_context_digest +
ordered_input_content_digests +
dependency_or_interface_digest
)
```
**Critical**: `snapshot_id` is NOT in the cache key. It goes in provenance only.
This permits reuse across commits, branches, and snapshots with identical units.
### Canonical Symbol IDs
**Native IDs** (when semantic backend is available):
- SCIP: `scip:{symbol}`
- Kythe: `kythe:{uri}`
- clangd: `clangd:{usr}`
**Fallback IDs** (when no semantic backend):
```
fallback:{language}:{file_path}:{sha256(qualified_name|start_line|signature)[:16]}
```
This distinguishes namespaces, overloads, methods, local functions, and
duplicate names across languages or repositories.
### Edge Kinds
Edge kinds are kept explicitly separate — never silently merged:
- `direct` — statically resolved call to a known symbol
- `indirect` — function pointer, closure, callback
- `virtual` — virtual method dispatch (runtime-resolved)
- `macro` — call introduced by macro expansion
- `unresolved` — callee name found but no symbol_id resolved
### Per-Language Semantic Units
| Language | Unit Type | Compile Context | Dependency Digest |
| ----------- | ---------------- | --------------------------- | ------------------------------------------- |
| C/C++ | Compilation unit | compile_commands.json entry | sha256 of transitive header interfaces |
| Go | Package | go.mod + build tags | sha256 of imported packages' exported API |
| Rust | Crate | Cargo.toml + features | sha256 of extern crate signatures |
| Java/Kotlin | Compiler batch | module path + classpath | sha256 of imported class signatures |
| TypeScript | Project | tsconfig.json | sha256 of imported module type declarations |
| Fallback | Individual file | None | Empty string |
**Fallback Granularity Rule:** For the fallback tier (regex, AST parser, or
lightweight crawler without a language build system), **each source file MUST be
its own independent semantic unit**. Never bundle multiple source files into a
single fallback unit. Per-file unit isolation is essential for content-addressed
cache efficiency — when one file changes in a future snapshot, only that single
file's unit invalidates while all unchanged files hit the cache
(`reused = N-1`).
### Baseline + Delta Overlay
- **Static baseline**: Built once in CI for the complete snapshot.
- **Incremental cache**: Immutable unit outputs reused across snapshots.
- **Delta overlay**: Changed/target files indexed locally, layered over
baseline.
- **Remote serving**: Optional when global index is too costly locally.
At query time, the query helper merges baseline + overlay results. Overlay
symbols supersede baseline symbols for the same file. The manifest records
`provider.kind = "baseline+overlay"` with baseline manifest reference and
overlay unit count.
### Compatibility JSONL Export
The old `structural_index.jsonl` format is retained as a compatibility export
only — it is NOT the query contract. Below the threshold (default 10K records),
a complete JSONL export is emitted. Above the threshold, only a provenance
pointer is written. Large consumers MUST use the query interface.
Line 1 — provenance header:
```json
{"_provenance": true, "snapshot_id": "abc123", "tool": "tree-sitter"}
```
Lines 2+ — structural records (one per line, `_type` discriminator):
```json
{"_type": "function", "key": "src/parser.c:parse_input", "start_line": 45, "end_line": 120, "signature": "int parse_input(char *buf, size_t len)", "calls": ["malloc", "validate_input", "memcpy"]}
{"_type": "call_edge", "caller": "parse_input", "callee": "malloc", "file": "src/parser.c", "line": 78}
{"_type": "call_edge", "caller": "main", "callee": "parse_input", "file": "src/main.c", "line": 203}
```
**Provenance header fields:**
| Field | Type | Description |
| ------------- | ------ | ----------------------------------------------------------------------------------- |
| `_provenance` | bool | Always `true` — marks this as the provenance header line |
| `snapshot_id` | string | `SNAPSHOT_ID` the index was built against |
| `tool` | string | Backend used (e.g. `"lsp-clangd"`, `"tree-sitter"`, `"ctags"`, `"regex"`, `"grep"`) |
**Record types:**
| `_type` | Description | Key fields |
| ----------- | ----------------------------------------------- | ----------------------------------------------------- |
| `function` | Function definition with boundary and signature | `key`, `start_line`, `end_line`, `signature`, `calls` |
| `call_edge` | A call from caller to callee at file:line | `caller`, `callee`, `file`, `line` |
## Snapshot Safety
1. **Build from `CODE_ROOT`, not live tree (when pinned).** When the snapshot is
pinned, the structural index is built from the pinned `CODE_ROOT`, ensuring
it reflects the exact bytes the pipeline is analyzing.
2. **Reuse-on-match.** If `manifest.json` already carries the current
`SNAPSHOT_ID` (and `SNAPSHOT_ID` is NOT `"unknown"` — in MODE-OFF, always
rebuild), reuse it — do not rebuild. Rebuild when `SNAPSHOT_ID` differs or is
`"unknown"`. Individual units may still be reused from the content-addressed
cache. This bounds cost across retries/crash-resume.
3. **Manifest atomicity.** The manifest is written LAST via atomic rename from
`tmp/`. Interrupted builds leave unreferenced temp objects without corrupting
the last published index. On resume, check `manifest.json` status and
`snapshot_id`.
4. **STALE flag in HALT mode.** When `snapshot_pinned` is false (HALT), the
index may be built from the unpinned `CODE_ROOT` but is marked as potentially
stale. Consumers treat structural hints as advisory.
5. **MODE-OFF: build, do not skip.** When `active_snapshot` is absent
(MODE-OFF), build against the current directory (cwd) with provenance
`snapshot_id` set to `"unknown"`. This is the standalone-efficiency case —
the index is still useful for the researcher/planner even without snapshot
pinning. Do NOT return an empty index merely because the snapshot is absent.
## Consumption Contract
These are runtime instructions for callers — they define how consumers use the
structural index. The structural index is a HINT-only enhancement; skills that
do not use it behave exactly as they do today.
### Query Interface
**The query helper IS the contract.** All consumers use
`query_structural_index.py` (or a compatible remote endpoint). The JSONL file is
NOT the query contract — it is a compatibility export only.
**Operations**:
1. `resolve_symbol(name, language?, file?, namespace?)` →
`{results, total, ambiguous, coverage}`
- Returns ALL matches — no silent selection among ambiguous symbols.
- Caller MUST disambiguate before calling `find_callers` / `find_callees`.
2. `find_callers(symbol_id, limit=100, offset=0)` →
`{results, total, has_more, coverage}`
- Bounded, paginated caller lookup.
- Each result carries `precision`, `backend`, `edge_kind`.
- Empty results carry `coverage.partition_status`.
3. `find_callees(symbol_id, limit=100, offset=0)` →
`{results, total, has_more, coverage}`
- Same shape as `find_callers`.
4. `get_function_boundary(file, line)` →
`{symbol_id, start_line, end_line, signature, precision, backend}`
5. `get_coverage(file?)` →
`{total_files, indexed_files, failed, deferred, partition_status, backends_used}`
**Key properties**:
- Bounded results and pagination (prevents loading entire index into memory).
- Explicit name resolution before graph traversal (no silent selection among
ambiguous symbols).
- Precision and backend attached to every result.
- Coverage attached to empty results.
- Identical operations for local and remote providers (remote via
`MANTIS_STRUCTURAL_INDEX_URL` env var or manifest `provider.kind = "remote"`).
**Coverage on empty results**:
| `partition_status` | Meaning | Consumer action |
| ------------------ | ----------------------------- | -------------------------------------------------------- |
| `complete` | All files indexed | "No indexed callers" (still run grep per HINT-only rule) |
| `partial` | Some files deferred or failed | "Not fully indexed" — MUST run grep fallback |
| `empty` | No backend available | "Not indexed" — MUST run grep fallback |
| `failed` | Backend attempted but failed | "Index failed" — MUST run grep fallback |
### mantis-plan
- Use the structural index query helper for function-level dependency fan-out.
When planning investigations, call `resolve_symbol()` then `find_callers()` to
identify all functions that call into a target — this broadens the audit set
beyond single-file analysis.
- The structural index decides ORDER of investigations (which functions to audit
first based on call-graph centrality), never MEMBERSHIP. It may broaden the
audit set (safe over-reporting), but must never REMOVE or drop a file; the
planner's existing logic remains the membership floor.
### mantis-researcher
- **Wave 1 (Rapid Triage):** Run a repo-wide grep for the function name to build
the exhaustive set of candidate call-sites — this is the mandatory floor. Then
use the structural index query helper (`resolve_symbol` then `find_callers`)
to RANK and prioritize which call-sites to audit first (the index
distinguishes actual calls from comments/strings/variable names). Audit the
union of both result sets — the structural index may miss macro-based calls,
function pointers, and dynamic dispatch, so grep remains the floor.
- **Wave 2 (Deep Audit):** Use `get_function_boundary(file, line)` to start with
the enclosing function, expanding to callers/callees/file as needed for
cross-function context — this saves context while preserving coverage.
- If the structural index is absent or empty, fall back to grep-based discovery
(today's behavior). The structural index is a coverage HINT only — it improves
audit quality but is never required.
## Safety
Non-negotiable invariants:
1. **Agnostic.** Nothing is ever required; no-tool / parse-fail / not-invoked →
empty index → grep fallback = today's behavior byte-for-byte. Optional in the
Pass Lifecycle Contract; a non-conformant harness simply skips it.
2. **HINT-only / union / never MEMBERSHIP.** Consumers audit the union with
grep; a symbol the index misses must still be reachable by the exhaustive
sweep. The structural index decides ORDER, never MEMBERSHIP. The query
contract enforces this: `find_callers` returns HINTs, never authoritative
membership — consumers MUST union with grep.
3. **No verdicts, touches no findings.** Cannot violate INV-1 and cannot itself
drop a finding.
4. **Narrow scope: source cross-reference only.** Binary/build-derived
reachability ("is it compiled into production") is explicitly out of scope —
a dev customization, not part of this skill, because absence-from-a-build can
hide a real finding (INV-2). Do not fold build-derived reachability into this
skill.
A reference blueprint is available at
[mantis-pipeline-adapter/references/mantis-structural-index.md](../mantis-pipeline-adapter/references/mantis-structural-index.md).
It is a stub that points to this SKILL.md as the single source of truth — do not
duplicate spec content there.