M2.5: symbol index and go-to commands

crates/index: walks the project with the search/VFS rules (vendor as
library), parses .php files from tree-sitter outlines on a worker pool
off the engine lock, updates incrementally from disk and buffers through
an updater thread, caps at 200,000 files, and answers fuzzy file, class
and symbol queries (nucleo-matcher, MPL-2.0 as a separate crate).

Engine: index.status, goto.file, goto.class, goto.symbol; events
indexProgress and indexReady drained by the pump; the index starts on
project.open and is cancelled on close. Tests: unit, incremental,
property (random queries and sources), engine and TCP control tests,
release timing on a generated 20k-file project (3.9-4.2 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
danny 2026-10-17 12:53:33 +02:00
parent f13a49064d
commit 053441d066
26 changed files with 3455 additions and 9 deletions

View file

@ -1,6 +1,6 @@
# Architecture
> **Status:** M0 decisions recorded, M1 workspace, desktop bridge, tabs and editor, message catalog, engine construction, settings, themes, the recovery journal, notifications, UI commands, the palette, the settings dialog, the app icon and the syntax layer documented, plus the search backend · **Last updated:** 2026-10-16
> **Status:** M0 decisions recorded, M1 workspace, desktop bridge, tabs and editor, message catalog, engine construction, settings, themes, the recovery journal, notifications, UI commands, the palette, the settings dialog, the app icon and the syntax layer documented, plus the search backend and the symbol index · **Last updated:** 2026-10-17
## Overview
@ -218,6 +218,19 @@ Watched directories: the VFS watcher is non-recursive, so `project.children` wat
- Engine: `search.find {query, regex?, caseSensitive?, wholeWord?, include?, exclude?, maxResults?}` (disabled without a project) snapshots the text of every dirty buffer as an overlay (searched instead of the file; only for files the walk visits), starts the search and returns `{searchId}` at once. Searches run outside the engine lock. Each search's results wait in its own channel in `State::searches`; `Engine::pump_vfs` drains them (`cmd::search::pump`) into `searchResults` events (one per file) and a final `searchDone`. Hosts already publish the pump's events, so the desktop (pump every 100 ms) and the control channel get them without new plumbing; there are at most 10 batches per second and at most 500 `searchResults` per pump (`MAX_RESULTS_PER_PUMP`), which keeps a control subscriber's queue of 4096 events from overflowing; the rest wait for the next pump. `search.cancel {searchId}` returns `{cancelled}` (false once the search ended) and the search still ends with `searchDone {cancelled: true}`. Starting a search does not cancel older ones. A search whose project is no longer open is cancelled by the next pump, its remaining results are dropped and it ends with `cancelled: true`.
- Measured (release, Windows 11, 16 threads, generated 20,006-file tree from `lutra_testkit::generate_search_tree`, other builds running): common literal `function` first result 10–43 ms, capped at 10,000 matches after 112–209 ms; a rare literal (full walk, 376 files) first result 17–22 ms, total 0.66–1.43 s (the first round cold); `function` with the 100,000 cap 0.53 s. Through the engine add up to one pump interval (100 ms) before results reach clients. Live check over the control channel against the running app (debug build, 5,900-file project with a dirty buffer): `search.find` answered in 4.5 ms, first `searchResults` line after 55 ms, `searchDone` after 386 ms; vendor hits present, `var/`, `node_modules` and the binary file absent; the dirty buffer's text was found; `search.cancel` ended a search with `cancelled: true`.
### Index and go-to (M2.5)
- `crates/index` (L2, depends on `lutra-syntax`, `lutra-text`, `ignore` and `nucleo-matcher` 0.3, MPL-2.0 as a separate crate, built without its `unicode-segmentation` feature) lists every file of the project and the declarations of its PHP files. A `Symbol` is `{name, fqn, kind, container, line, column}`: names as written (`$id` for properties), FQNs without a leading `\` (`App\Models\User`, members `App\Models\User::find`), `container` the declaring class-like FQN, the position of the name (0-based line, UTF-16 column) from the outline's `selectionRange`. Namespaces only prefix FQNs; anonymous classes and their members are skipped; at most 20,000 symbols per file.
- Which files: the walk is configured to the same rules as `crates/search` (ignore files without a git repository, `.git` and `node_modules` skipped, the root's `vendor/` walked as a second root even when gitignored and marked `library`, dot files listed, links not followed). Every file is listed for Go to File; only `.php` files up to 2 MiB (`MAX_PARSE_BYTES`) are parsed, larger ones are listed without symbols. At most 200,000 files (`MAX_FILES`); further files are left out (`Summary::capped`).
- Build: `build`/`spawn_build` walk first (parallel `ignore` walker), report non-PHP files at once, then parse PHP files on `default_threads()` workers (cores − 1, at most 32), each with its own `Parser` and one file in memory, taking files from a shared counter: project files before library files, larger files first in each group. Workers send `Update::Files` batches of 64 files (or every 100 ms) and `Update::Progress {done, total}`; the build ends with `Update::Ready(Summary)`. A panic while parsing a file (a parser bug) is caught per file: the file is listed without symbols and the worker gets a fresh parser. Reads are capped, invalid UTF-8 is replaced, a BOM dropped.
- Incremental: `Updater` is one background thread taking `Job::File {path, text?}` (re-index from a buffer's text, or re-read from disk; a file no longer found or now excluded is reported `Removed`) and `Job::Dir {path}` (drop the subtree, list it again with a walk restricted to the path, so ignore files on the way still apply). Jobs queued while it works are coalesced to the last per path, so a burst of keystrokes parses a buffer once.
- Queries: `Index` (a `BTreeMap` by path, so a directory's files drop with one range) answers `find_files`, `find_classes` and `find_symbols`. Every whitespace-separated query word must match fuzzily (fzf-style scoring, smart case, no other special syntax); queries are capped at 256 characters and 1,000 results. Files: a query without `/` (a `\` counts as `/`) is matched against the file name first with double weight, else the path; match ranges are merged UTF-16 ranges in the path. Classes: by name, by FQN when the query has a `\`. Symbols: by name; `Class::member` matches the container's short name (FQN with a `\`) and the member separately, `Class::` lists members. Order: project before library, higher score, shorter name, byte order. Indexes over 20,000 files plus symbols are scanned on `default_threads()` threads, each keeping its own best hits.
- Engine (`cmd/goto.rs`): `State::index` holds the index of the open project. `goto::sync` runs on every `Engine::pump_vfs` and before every index command: it starts the build and the updater when a project is open and the index belongs to another root (or none), drops them without a project (dropping cancels the threads; `project.close` also drops it at once), merges up to 4,096 updates, emits `indexProgress {done, total}` at most every 250 ms while building and `indexReady {files, symbols, elapsedMs}` at the end, and re-indexes buffers. The engine lock is held only for inserts and queries; walking and parsing never run under it. Updater results that arrive while the build runs mark their paths, and later build results for those paths are ignored, so a buffer edit or a disk change is never overwritten by the stale build.
- Incremental sources: `workspace::pump` hands the VFS events to `goto::note_disk_changes` (removed paths dropped at once, created or modified files re-read, created directories and rename targets rescanned); the watcher is not recursive, so only watched directories (root, listed, pinned) report changes. A buffer whose version changed is re-indexed from its text when its path is in the index (or the build still runs); while it is dirty, disk changes for it are ignored; closing it dirty re-reads the file.
- Commands: `index.status` → `{state: "idle"|"indexing", files, symbols}` (idle and zero without a project). `goto.file {query, limit?}` → `{items: [{path, score, matches: [[start, end]]}], indexing}`; `goto.class` → `{items: [{name, fqn, kind, path, line, column, library}], indexing}`; `goto.symbol` → the same plus `container` (FQN or `null`), for functions, methods, properties, constants and enum cases. `limit` defaults to 50 (at most 1,000); an empty query returns no items. The go-to commands are disabled without a project; none has a menu path or shortcut yet (M2.6 adds the popup).
- Measured (release, Windows 11, 8 cores / 16 threads, 15 workers, generated `lutra_testkit::generate_php_project(20_000)`: 20,001 files, 19,000 PHP of ~5 KB, 87.7 MB, 324,080 symbols): build 3.9–4.2 s on an idle machine (target < 5 s met), 4.6–5.7 s while other agents' builds ran; the walk takes 25–35 ms and the first batch arrives after ~30 ms, so the parse dominates (~21 MB/s over 15 workers, ~1.4 MB/s per worker). Queries on the full index, median of 7: `goto.class` 2–4 ms, `goto.symbol` 7–11 ms, `Class::member` 11–12 ms, `goto.file` 2 ms (single-threaded before the parallel scan: `goto.symbol` 72–100 ms). `cargo test --release -p lutra-index --test perf -- --ignored --nocapture` repeats it.
- Live check over the control channel against the running app (debug build, port 7825, generated 6,002-file project with gitignored `vendor/` and `var/`): `index.status` answered `indexing` right after `project.open`, a go-to query during the build returned `indexing: true`, 25 `indexProgress` events at least 251 ms apart, `indexReady` after 8.2 s (debug build: the tree-sitter C code and the workers are unoptimised) with 6,002 files and 62,000 symbols; `goto.file`, `goto.class` (vendor marked `library`), `goto.symbol` with `Class::member` answered in 5–25 ms; the gitignored `var/` class was absent; a `buffer.edit` renaming a class and a file created on disk were found within a second; `project.close` reset the status to idle.
## Decisions (M0)
| Question | Decision | Why |
@ -235,6 +248,7 @@ Watched directories: the VFS watcher is non-recursive, so `project.children` wat
| Date | Change | Summary |
|---|---|---|
| 2026-10-17 | minor | M2.5: symbol index (`crates/index`: parallel build, updater, fuzzy queries), `index.status`, `goto.file`/`goto.class`/`goto.symbol`, `indexProgress`/`indexReady` via the pump, timing |
| 2026-10-16 | minor | M2.3: find in files backend (`crates/search`, `search.find`/`search.cancel`, `searchResults`/`searchDone` via the pump) |
| 2026-10-16 | minor | M2.1: syntax layer (tree-sitter PHP parser, incremental reparse, outline, folding, size cap, timing) |
| 2026-10-16 | minor | M1.15: modal rule (one modal at a time, restore dialog queued), `recoveryResolved` event |

View file

@ -1,6 +1,6 @@
# Gaps
> **Status:** M0 closed G1; G2 under way (text buffer, engine buffers, tabs and editor in the app, recovery journal); G3 M1 part done (project tree, palette, settings dialog, themes); G4 under way (tree-sitter outline and folding library); G14 English UI catalog in place · **Last updated:** 2026-10-16
> **Status:** M0 closed G1; G2 under way (text buffer, engine buffers, tabs and editor in the app, recovery journal); G3 M1 part done (project tree, palette, settings dialog, themes); G4 under way (tree-sitter outline and folding library); G14 English UI catalog in place · **Last updated:** 2026-10-17
Every known shortfall, ranked. This is the work list: unless the owner gives a task, take the highest open gap. Hours are agent wall-clock hours for one agent, all estimated.
@ -31,7 +31,7 @@ G11 progress: M1.7 added the crash-recovery journal: unsaved buffers are written
G14 progress: M1.11 moved every UI string in `ui/src` into the English catalog `ui/src/i18n/en.json` behind `t(key, params?)`, enforced by `vue/no-bare-strings-in-template` and a catalog test. Messages the engine sends (`EngineError.message`, the slogan from `app.about`) are still English text; they need message keys with params before a second language can cover them.
G5 progress: M2.3 landed the find-in-files backend: `crates/search` (parallel walk with the project's ignore rules, literal or regex, case, whole word, globs, binary and size skips, match cap, cancel) and the engine commands `search.find` / `search.cancel` with streamed `searchResults` and `searchDone` events; dirty buffers are searched in their buffer text. First result on a generated 20k-file tree 10–43 ms (release, measured). Open in G5: the Find tool window (M2.4), replace in files, the symbol index and go-to (M2.5, M2.6).
G5 progress: M2.3 landed the find-in-files backend: `crates/search` (parallel walk with the project's ignore rules, literal or regex, case, whole word, globs, binary and size skips, match cap, cancel) and the engine commands `search.find` / `search.cancel` with streamed `searchResults` and `searchDone` events; dirty buffers are searched in their buffer text. First result on a generated 20k-file tree 10–43 ms (release, measured). M2.5 added the first symbol index (`crates/index`: every project file, PHP declarations from tree-sitter outlines, vendor as library, parallel build off the engine lock, incremental on disk and buffer changes, cancelled on project close) and the commands `index.status`, `goto.file`, `goto.class`, `goto.symbol` with `indexProgress`/`indexReady` events; a generated 20k-file project (88 MB of PHP) indexes in 3.9–4.2 s (release, measured, target 5 s), queries take 2–12 ms. Open in G5: the Find tool window (M2.4), replace in files, the go-to popup (M2.6), an index that survives restarts, changes in directories the non-recursive watcher does not watch, usages.
Known smaller items from M0:
@ -43,6 +43,7 @@ Known smaller items from M0:
| Date | Change | Summary |
|---|---|---|
| 2026-10-17 | minor | G5: symbol index and go-to commands (M2.5) |
| 2026-10-16 | minor | G5: find-in-files backend (M2.3) |
| 2026-10-16 | minor | G4 note: tree-sitter syntax library, outline and folding (M2.1) |
| 2026-10-16 | minor | G3: settings dialog, theme picker (M1.9b) |

View file

@ -1,6 +1,6 @@
# Target-app parity
> **Status:** M1 done (text buffer, VFS, engine workspace, project tree, tabs and a plain-text editor, recovery, palette, settings and themes in the app), no PHP feature ships · **Last updated:** 2026-10-16 · **Target:** PhpStorm 2026.x
> **Status:** M1 done (text buffer, VFS, engine workspace, project tree, tabs and a plain-text editor, recovery, palette, settings and themes in the app), no PHP feature ships · **Last updated:** 2026-10-17 · **Target:** PhpStorm 2026.x
## How this is measured
@ -14,7 +14,7 @@ Hours are the estimated agent wall-clock hours for one agent to reach behaviour
|---|---:|---:|---|---:|---|
| Editor core | 12 | 9% (est.) | syntax highlighting (M2), code folding, soft wrap and other editor settings, preview tab, pinned/split/reordered tabs, close others/all, reopen closed tab, tab limit, conflict UI for files changed on disk, autosave, local history, encodings other than UTF-8, large-file mode, column selection and multi-caret checked against the docs; recovery journal not yet checked in the running app (gaming mode) | ~106 (est.) | M2.1: `crates/syntax` parses PHP with tree-sitter (incremental reparse from `lutra_text` edits) and computes folding ranges (brace blocks, arrays, comments, heredocs, `use` runs); library only, nothing visible in the editor yet, so the score is unchanged. Tests: outline/folding snapshots for five PHP 8.3 fixtures, proptests (random bytes and token soups never panic, reparse equals a fresh parse, undo round trip). 1 MB parse 364–553 ms, reparse after one keystroke 7–11 ms (release, measured, machine busy). M1.13: opening a file that is already open switches to its tab (`bufferFocused`; engine test, Vitest, control-channel test; live check pending (gaming mode)); files in folders the tree never expanded still reload or report changes on disk, because the parent directory of every open buffer stays watched (engine tests). M1.7: crash-recovery journal (`crates/engine/src/journal.rs`, `cmd/recovery.rs`): dirty buffers are written atomically to `<data_dir>/recovery/<project-id>/` about 1 s after the last edit (at least every 2 s while typing), never per keystroke; removed on save, close and discard; `recoveryAvailable` after `project.open`, `recovery.list/restore/discard`; restored buffers are dirty and keep line ending and BOM; corrupt entries are moved aside with a warning. Restore dialog (`ui/src/components/recovery/`: Restore all / selected / Discard, time, changed-on-disk hint). Tests: engine dropped mid-edit and a new engine restores the exact CRLF + BOM text, corrupt entries, removal paths, a property test of random edit sequences → crash → restore equals the buffer (`crates/engine/tests/recovery.rs`, 10 tests), the same flow headless over the control channel (`crates/control/tests/recovery.rs`), 10 Vitest tests for the dialog. 1 MiB entry write: 8.5 ms median (release, measured). Graded against `plan/phpstorm/04-recovery.md`. M1.6: tabs and a CodeMirror 6 editor in the app (`ui/src/components/editor/`, `ui/src/stores/editor.ts`): open from the project tree or engine events, activate, close by button, middle click or `Mod+W`, dirty dot, Save / Don't Save / Cancel on closing a dirty tab, `Mod+S` / `Mod+Alt+S` from engine command specs, non-blocking notice for a dirty file changed or deleted on disk; local edits go to `buffer.edit` with version-based resync, CRLF files keep `\r\n`; undo/redo, multiple selections, search and bracket matching from CodeMirror's basic setup. Vitest: a burst of 400 keystrokes with stale versions and remote edits loses none, a CRLF burst with deletions converges, store and component tests against a fake engine (`ui/tests/editor/`, 22 tests). Checked in the app on Windows 11 with `--control` and `ui.screenshot`: typing via the UI and via `buffer.edit`, save to disk, close question, middle click, disk-change notice. 1 MB file (16,385 lines): engine `file.open` 80 ms over the control channel (debug build, measured), editor state 1–6 ms and first rendered view 4–11 ms in Chromium (measured), ≈ 90 ms in total (estimated, budget 150 ms). Graded against `plan/phpstorm/03-editor-tabs.md`. `crates/text`: rope buffer, positions, edits (M1.1, unit tests + proptests). M1.3: engine buffers with versions and dirty state, `file.open`, `buffer.get`, `buffer.edit` (UTF-16 change sets, `stale_version`), `file.save`, `file.saveAll`, `buffer.close` with unsaved-changes guard, line ending and BOM kept on save, clean buffers reload on external change; 14 integration tests on temp dirs (`crates/engine/tests/workspace.rs`) |
| Project view, VFS | 6 | 8% (est.) | views list (project / scopes), toolbar (Select Opened File, Expand/Collapse All), sort and display options, show/hide excluded, distinct icons for library and excluded folders, file colours, speed search, preview, file operations | ~37 (est.) | `lutra-vfs` (M1.2): lazy sorted listing, ignore rules (.gitignore, .ignore, info/exclude, global excludes, Lutra defaults), native file watching via `notify` with a polling fallback, capped UTF-8 reads, atomic writes; 34 tests on temp dirs, watcher tests run on both backends (`crates/vfs/tests`). M1.5: project tree UI (lazy listing, keyboard navigation, dimmed ignored entries, event refresh) in `ui/src/components/project/`, Vitest against a mocked engine. M1.3: `project.open`, `project.close`, `project.children` (watches listed directories, capped at 256 native / 32 polling, least recently listed dropped), `treeChanged` events from disk, `events.subscribe` on the control channel with a TCP test (`crates/control/tests/events.rs`). M1.4: the tree runs against the engine in the app: folder picker (`project.openDialog`) and `lutra-desktop --control PORT PATH` open a project, the tree lists it lazily with ignored (`var/` from `.gitignore`) and excluded (`node_modules`) entries dimmed, a file created on disk appears in an expanded folder within ~100 ms; verified with `ui.screenshot` on Windows 11 and Vitest (`ui/tests/project/`, `ui/tests/ipcEvents.test.ts`). M1.13: a directory listed only partly (more than 100,000 entries) ends with a "Listing truncated" note row (Vitest). M1.12: a folder that cannot be opened from the picker or the command line (including `unsaved_changes` when switching projects) and stopped file watching show as toasts instead of stderr (`apps/lutra-desktop/src/host.rs`, `ui/src/stores/notifications.ts`, `ui/tests/notifications/`, 19 Vitest tests; live check pending, gaming mode); score unchanged. Graded against `plan/phpstorm/02-project-view.md`: tool window, folder-first tree, dimmed ignored/excluded, keyboard navigation and live refresh exist; most toolbar, view and display options do not |
| Navigation | 10 | 0% | everything (M1.8: a command palette over commands only, no go-to-file/class/symbol yet; score unchanged) | ~120 (est.) | M2.1: file outline in `crates/syntax` (namespaces braced and unbraced, classes, interfaces, traits, enums with cases, functions, methods, properties incl. promoted ones, constants; UTF-16 ranges and short signatures), the base for the Structure window (M2.7) and the symbol index (M2.5); not reachable from the UI yet, score unchanged. M1.8: `ui/src/components/palette/`, `ui/src/commands/` (fuzzy ranking, recent commands, global keymap), Vitest `ui/tests/palette/`, `ui/tests/commands/` |
| Navigation | 10 | 0% | everything a user can reach: the go-to popup (M2.6), Structure (M2.7), find usages, go to declaration, recent files, navigation history (M1.8: a command palette over commands only; M2.5: go-to backend without UI; score unchanged) | ~112 (est.) | M2.5: symbol index in `crates/index` (every project file listed, `.php` parsed in parallel from tree-sitter outlines, vendor marked library, incremental on disk and buffer changes, capped at 200,000 files) and the engine commands `index.status`, `goto.file`, `goto.class`, `goto.symbol` (fuzzy, file names weighted, `Class::member`, project before library, partial results with `indexing: true`), events `indexProgress`/`indexReady`; 20k-file / 88 MB generated project indexed in 3.9–4.2 s (release, measured), queries 2–12 ms; tests `crates/index/tests/`, `crates/engine/tests/goto.rs`, `crates/control/tests/goto.rs`; not reachable from the UI yet (M2.6). M2.1: file outline in `crates/syntax` (namespaces braced and unbraced, classes, interfaces, traits, enums with cases, functions, methods, properties incl. promoted ones, constants; UTF-16 ranges and short signatures), the base for the Structure window (M2.7) and the symbol index (M2.5); not reachable from the UI yet, score unchanged. M1.8: `ui/src/components/palette/`, `ui/src/commands/` (fuzzy ranking, recent commands, global keymap), Vitest `ui/tests/palette/`, `ui/tests/commands/` |
| Search and replace | 6 | 0% (est.) | everything a user sees: Find in Files tool window (M2.4), replace in files, scopes, file masks UI, search history, find in path from the tree, structural search; the backend below has no UI yet, so the score stays 0% | ~76 (est.) | M2.3 backend: `crates/search` (parallel `ignore` walk + `grep-searcher`: literal/regex, case, whole word, include/exclude globs, project ignore rules with `vendor/` searched and `node_modules` skipped, binary and > 8 MiB files skipped, match cap, cancel) and engine `search.find` / `search.cancel` with `searchResults` / `searchDone` events through the pump; dirty buffers are searched in their buffer text. Tests: 9 library tests incl. a property test of random queries and file bytes (`crates/search/tests/search.rs`), 3 preview/column unit tests, 4 engine tests (`crates/engine/tests/search.rs`), a TCP control-channel test (find → results → done, invalid regex, cancel; `crates/control/tests/search.rs`). 20k-file generated tree, release: first result 10–43 ms, full walk 0.66–1.43 s (measured); live over the control channel against the running app: first result 55 ms, done 386 ms on a 5,900-file project (debug build, measured) |
| Code completion | 10 | 0% | everything | ~250 (est.) | — |
| Inspections and quick-fixes | 8 | 0% | everything | ~300 (est.) | — |
@ -63,6 +63,7 @@ The project tree is the first user-visible feature (Project view 8%, estimated);
| Date | Change | Summary |
|---|---|---|
| 2026-10-17 | minor | M2.5: Navigation evidence (symbol index and go-to commands, no UI), score unchanged at 0%, hours 120 → 112 |
| 2026-10-16 | minor | M2.3: Search and replace evidence (find-in-files backend, no UI), score unchanged at 0%, hours 80 → 76 |
| 2026-10-16 | minor | M2.1: Editor core and Navigation evidence (tree-sitter parse, outline, folding library), scores unchanged |
| 2026-10-16 | minor | M1.9b: Settings row 4% → 7% (est.), settings dialog; M1 done |