M2.1: tree-sitter PHP in crates/syntax

Parser (parse, incremental reparse from lutra_text edits), outline with
UTF-16 ranges and short signatures, folding ranges, 8 MiB parse cap.
Snapshot tests for five PHP 8.3 fixtures, proptests (random bytes,
reparse equals fresh parse, undo round trip), ignored release timing
test. Docs: architecture Syntax section, parity evidence, G4 note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
danny 2026-10-16 22:06:21 +02:00
parent d12a517009
commit a5236acd7e
25 changed files with 1768 additions and 6 deletions

View file

@ -1,6 +1,6 @@
# Architecture
> **Status:** M0 decisions recorded, M1 workspace, desktop bridge, tabs and editor, message catalog, engine construction, settings, themes, the recovery journal, notifications, UI commands, the palette, the settings dialog and the app icon documented · **Last updated:** 2026-10-16
> **Status:** M0 decisions recorded, M1 workspace, desktop bridge, tabs and editor, message catalog, engine construction, settings, themes, the recovery journal, notifications, UI commands, the palette, the settings dialog, the app icon and the syntax layer documented · **Last updated:** 2026-10-16
## Overview
@ -194,6 +194,18 @@ Watched directories: the VFS watcher is non-recursive, so `project.children` wat
- **Keyboard:** Escape or a click outside closes; Tab and Shift+Tab wrap inside the dialog, focus that leaves it is pulled back to the search field, and focus returns to where it was on close.
- No OK/Apply/Cancel: every change applies at once, as agents change settings through the same commands.
### Syntax (M2.1)
`crates/syntax` (L1, depends on `lutra-text` only) parses PHP with tree-sitter (`tree-sitter` 0.26, `tree-sitter-php` 0.25, both MIT; the grammar variant that accepts inline HTML around `<?php` tags). It has no engine types; M2.7 wires it to buffers and M2.5 builds the index on it.
- **API:** `Parser::new()?`, `parser.parse(&str) -> Tree` (never fails), `parser.reparse(&Tree, &[lutra_text::Edit]) -> Result<Tree, SyntaxError>`, `outline(&Tree) -> Vec<OutlineItem>`, `folding(&Tree) -> Vec<Range>`. `Tree` carries the text it was parsed from (a `lutra_text::Buffer`, cheap to clone) plus `truncated()` and `has_errors()`; that is why `outline` takes no separate text argument (the contract sketch had `outline(tree, text)`): a tree and a mismatching text can't be combined.
- **Reparse:** the edit batch has `Buffer::apply` semantics (ranges against the old text, any order). It is mapped to one tree-sitter `InputEdit` covering the smallest start to the largest end, with points (row, byte column, rows by `\n` as tree-sitter counts them) computed from the old and new text. Proptests check that a reparse gives the same tree, outline and folding as a fresh parse whenever the edited text is error-free, and that an undo round trip ends at the original tree. On broken code tree-sitter's error recovery may legitimately differ between an incremental and a fresh parse; the engine can reparse from scratch when it wants a canonical tree (for example on save).
- **Outline:** `OutlineItem {kind, name, detail, range, selection_range, children}`; `OutlineKind::as_str()` gives the contract names (`namespace`, `class`, …, `enumCase`). Braced and unbraced namespaces (an unbraced one owns everything up to the next namespace statement), promoted constructor parameters as properties after the constructor, enum cases with backing values, multi-element `const`/property declarations as one item each, attributes inside the declaration range. Anonymous classes are classes named `class@anonymous` (`OutlineItem::is_anonymous`, for the index to skip). Function, method and closure bodies are not searched; statements at file level are (`if (!function_exists(…)) { function … }`). Error nodes are transparent and nameless declarations are dropped. The search descends at most 96 levels.
- **Folding:** multi-line brace blocks (class-like bodies, function and control bodies, `switch`/`match`), arrays, comments, heredoc/nowdoc, `use A\{…}` groups and runs of two or more `use` statements; ranges include the delimiters. The walk is iterative and skips the inside of single-line nodes.
- **Positions:** 0-based line and UTF-16 column through `Buffer::offset_to_position` (about 1 µs per conversion on a 1 MB rope; the parse dominates).
- **Limits:** texts over 8 MiB (`MAX_PARSE_BYTES`) are not parsed (`truncated`, empty outline and folding). The grammar's identifier rule covers the Basic Multilingual Plane only: PHP accepts any byte ≥ 0x80 in names, so an identifier with an emoji is an error node in tree-sitter.
- **Timing** (release, Windows 11, MSVC `/O2`, generated 1 MB file, best of 5, other builds running in parallel, measured): full parse 364–553 ms, reparse after one inserted character 7–11 ms, outline 70–133 ms, folding 61–73 ms (143 ms before skipping single-line nodes). The full parse misses the 50 ms target by about 8×: tree-sitter-php runs at roughly 2–3 MB/s here. Consequences: parse off the engine lock and debounce (M2.7), parse files in parallel for the index (M2.5); `cargo test --release -p lutra-syntax --test perf -- --ignored --nocapture` repeats the measurement.
## Decisions (M0)
| Question | Decision | Why |
@ -211,6 +223,7 @@ Watched directories: the VFS watcher is non-recursive, so `project.children` wat
| Date | Change | Summary |
|---|---|---|
| 2026-10-16 | minor | M2.1: syntax layer (tree-sitter PHP parser, incremental reparse, outline, folding, size cap, timing) |
| 2026-10-16 | minor | M1.9b: settings dialog (store, search, controls by type, reset, live updates, font-size token override) |
| 2026-10-16 | minor | M1.14: app icon from the owner's master, multi-size ICO, placeholder as fallback |
| 2026-10-16 | minor | M1.8: UI commands (`ui.registerCommands`, `uiCommand`, `ui` in listCommands), global keymap, command palette |

View file

@ -1,6 +1,6 @@
# Gaps
> **Status:** M0 closed G1; G2 under way (text buffer, engine buffers, tabs and editor in the app, recovery journal); G3 M1 part done (project tree, palette, settings dialog, themes); G14 English UI catalog in place · **Last updated:** 2026-10-16
> **Status:** M0 closed G1; G2 under way (text buffer, engine buffers, tabs and editor in the app, recovery journal); G3 M1 part done (project tree, palette, settings dialog, themes); G4 under way (tree-sitter outline and folding library); G14 English UI catalog in place · **Last updated:** 2026-10-16
Every known shortfall, ranked. This is the work list: unless the owner gives a task, take the highest open gap. Hours are agent wall-clock hours for one agent, all estimated.
@ -25,6 +25,8 @@ Every known shortfall, ranked. This is the work list: unless the owner gives a t
G2 progress: the engine text buffer landed (M1.1, `crates/text`: rope, UTF-16 and char positions, atomic edit batches, line ending policy). M1.3 added engine buffers with versions, the edit protocol, save and save all, and reloading on external changes. M1.6 added tabs and a CodeMirror 6 editor synced to engine buffers (version-based edits with resync, CRLF-aware offsets), save and save all by engine shortcuts, the Save / Don't Save / Cancel question and a notice for files changed on disk. M1.7 added the crash-recovery journal (debounced atomic entries per dirty buffer, `recovery.*` commands, restore dialog; not yet checked in the running app). M1.13 made re-opening an open file switch to its tab and keeps the folders of open files watched. Still open: tab management beyond open/close, a conflict UI (compare and merge with the disk version), autosave, editor settings (font size, soft wrap); estimate lowered to ~16 h.
G4 progress: M2.1 added the structural half in `crates/syntax`: tree-sitter PHP parse and incremental reparse, outline (namespaces, class-likes, functions, members, promoted properties, enum cases; anonymous classes marked) and folding ranges, an 8 MiB parse cap, proptests and snapshots. Not visible yet: highlighting (M2.2) and the Structure window (M2.7) are open. Finding: a full parse of 1 MB takes 364–553 ms on the dev machine (target 50 ms; reparse 7–11 ms), so parsing must stay off the engine lock and the index must parse in parallel.
G11 progress: M1.7 added the crash-recovery journal: unsaved buffers are written atomically about 1 s after the last edit and offered back after a crash; corrupt entries are skipped with a warning. Covered by a property test (random edits, crash, restore equals the buffer) and a headless control-channel test; not yet checked by killing the running app (gaming mode). Open: journal writes run under the engine lock (8.5 ms for 1 MiB, release), two instances on one project share a journal, local history.
G14 progress: M1.11 moved every UI string in `ui/src` into the English catalog `ui/src/i18n/en.json` behind `t(key, params?)`, enforced by `vue/no-bare-strings-in-template` and a catalog test. Messages the engine sends (`EngineError.message`, the slogan from `app.about`) are still English text; they need message keys with params before a second language can cover them.
@ -39,6 +41,7 @@ Known smaller items from M0:
| Date | Change | Summary |
|---|---|---|
| 2026-10-16 | minor | G4 note: tree-sitter syntax library, outline and folding (M2.1) |
| 2026-10-16 | minor | G3: settings dialog, theme picker (M1.9b) |
| 2026-10-16 | minor | G3: command palette, global keymap, UI commands (M1.8) |
| 2026-10-16 | minor | G2/G3: re-open focuses the tab, open buffers' folders watched, truncated listings shown (M1.13) |

View file

@ -12,9 +12,9 @@ Hours are the estimated agent wall-clock hours for one agent to reach behaviour
| Area | Weight | Score | Missing | Hours (one agent) | Evidence |
|---|---:|---:|---|---:|---|
| Editor core | 12 | 9% (est.) | syntax highlighting (M2), code folding, soft wrap and other editor settings, preview tab, pinned/split/reordered tabs, close others/all, reopen closed tab, tab limit, conflict UI for files changed on disk, autosave, local history, encodings other than UTF-8, large-file mode, column selection and multi-caret checked against the docs; recovery journal not yet checked in the running app (gaming mode) | ~106 (est.) | M1.13: opening a file that is already open switches to its tab (`bufferFocused`; engine test, Vitest, control-channel test; live check pending (gaming mode)); files in folders the tree never expanded still reload or report changes on disk, because the parent directory of every open buffer stays watched (engine tests). M1.7: crash-recovery journal (`crates/engine/src/journal.rs`, `cmd/recovery.rs`): dirty buffers are written atomically to `<data_dir>/recovery/<project-id>/` about 1 s after the last edit (at least every 2 s while typing), never per keystroke; removed on save, close and discard; `recoveryAvailable` after `project.open`, `recovery.list/restore/discard`; restored buffers are dirty and keep line ending and BOM; corrupt entries are moved aside with a warning. Restore dialog (`ui/src/components/recovery/`: Restore all / selected / Discard, time, changed-on-disk hint). Tests: engine dropped mid-edit and a new engine restores the exact CRLF + BOM text, corrupt entries, removal paths, a property test of random edit sequences → crash → restore equals the buffer (`crates/engine/tests/recovery.rs`, 10 tests), the same flow headless over the control channel (`crates/control/tests/recovery.rs`), 10 Vitest tests for the dialog. 1 MiB entry write: 8.5 ms median (release, measured). Graded against `plan/phpstorm/04-recovery.md`. M1.6: tabs and a CodeMirror 6 editor in the app (`ui/src/components/editor/`, `ui/src/stores/editor.ts`): open from the project tree or engine events, activate, close by button, middle click or `Mod+W`, dirty dot, Save / Don't Save / Cancel on closing a dirty tab, `Mod+S` / `Mod+Alt+S` from engine command specs, non-blocking notice for a dirty file changed or deleted on disk; local edits go to `buffer.edit` with version-based resync, CRLF files keep `\r\n`; undo/redo, multiple selections, search and bracket matching from CodeMirror's basic setup. Vitest: a burst of 400 keystrokes with stale versions and remote edits loses none, a CRLF burst with deletions converges, store and component tests against a fake engine (`ui/tests/editor/`, 22 tests). Checked in the app on Windows 11 with `--control` and `ui.screenshot`: typing via the UI and via `buffer.edit`, save to disk, close question, middle click, disk-change notice. 1 MB file (16,385 lines): engine `file.open` 80 ms over the control channel (debug build, measured), editor state 1–6 ms and first rendered view 4–11 ms in Chromium (measured), ≈ 90 ms in total (estimated, budget 150 ms). Graded against `plan/phpstorm/03-editor-tabs.md`. `crates/text`: rope buffer, positions, edits (M1.1, unit tests + proptests). M1.3: engine buffers with versions and dirty state, `file.open`, `buffer.get`, `buffer.edit` (UTF-16 change sets, `stale_version`), `file.save`, `file.saveAll`, `buffer.close` with unsaved-changes guard, line ending and BOM kept on save, clean buffers reload on external change; 14 integration tests on temp dirs (`crates/engine/tests/workspace.rs`) |
| Editor core | 12 | 9% (est.) | syntax highlighting (M2), code folding, soft wrap and other editor settings, preview tab, pinned/split/reordered tabs, close others/all, reopen closed tab, tab limit, conflict UI for files changed on disk, autosave, local history, encodings other than UTF-8, large-file mode, column selection and multi-caret checked against the docs; recovery journal not yet checked in the running app (gaming mode) | ~106 (est.) | M2.1: `crates/syntax` parses PHP with tree-sitter (incremental reparse from `lutra_text` edits) and computes folding ranges (brace blocks, arrays, comments, heredocs, `use` runs); library only, nothing visible in the editor yet, so the score is unchanged. Tests: outline/folding snapshots for five PHP 8.3 fixtures, proptests (random bytes and token soups never panic, reparse equals a fresh parse, undo round trip). 1 MB parse 364–553 ms, reparse after one keystroke 7–11 ms (release, measured, machine busy). M1.13: opening a file that is already open switches to its tab (`bufferFocused`; engine test, Vitest, control-channel test; live check pending (gaming mode)); files in folders the tree never expanded still reload or report changes on disk, because the parent directory of every open buffer stays watched (engine tests). M1.7: crash-recovery journal (`crates/engine/src/journal.rs`, `cmd/recovery.rs`): dirty buffers are written atomically to `<data_dir>/recovery/<project-id>/` about 1 s after the last edit (at least every 2 s while typing), never per keystroke; removed on save, close and discard; `recoveryAvailable` after `project.open`, `recovery.list/restore/discard`; restored buffers are dirty and keep line ending and BOM; corrupt entries are moved aside with a warning. Restore dialog (`ui/src/components/recovery/`: Restore all / selected / Discard, time, changed-on-disk hint). Tests: engine dropped mid-edit and a new engine restores the exact CRLF + BOM text, corrupt entries, removal paths, a property test of random edit sequences → crash → restore equals the buffer (`crates/engine/tests/recovery.rs`, 10 tests), the same flow headless over the control channel (`crates/control/tests/recovery.rs`), 10 Vitest tests for the dialog. 1 MiB entry write: 8.5 ms median (release, measured). Graded against `plan/phpstorm/04-recovery.md`. M1.6: tabs and a CodeMirror 6 editor in the app (`ui/src/components/editor/`, `ui/src/stores/editor.ts`): open from the project tree or engine events, activate, close by button, middle click or `Mod+W`, dirty dot, Save / Don't Save / Cancel on closing a dirty tab, `Mod+S` / `Mod+Alt+S` from engine command specs, non-blocking notice for a dirty file changed or deleted on disk; local edits go to `buffer.edit` with version-based resync, CRLF files keep `\r\n`; undo/redo, multiple selections, search and bracket matching from CodeMirror's basic setup. Vitest: a burst of 400 keystrokes with stale versions and remote edits loses none, a CRLF burst with deletions converges, store and component tests against a fake engine (`ui/tests/editor/`, 22 tests). Checked in the app on Windows 11 with `--control` and `ui.screenshot`: typing via the UI and via `buffer.edit`, save to disk, close question, middle click, disk-change notice. 1 MB file (16,385 lines): engine `file.open` 80 ms over the control channel (debug build, measured), editor state 1–6 ms and first rendered view 4–11 ms in Chromium (measured), ≈ 90 ms in total (estimated, budget 150 ms). Graded against `plan/phpstorm/03-editor-tabs.md`. `crates/text`: rope buffer, positions, edits (M1.1, unit tests + proptests). M1.3: engine buffers with versions and dirty state, `file.open`, `buffer.get`, `buffer.edit` (UTF-16 change sets, `stale_version`), `file.save`, `file.saveAll`, `buffer.close` with unsaved-changes guard, line ending and BOM kept on save, clean buffers reload on external change; 14 integration tests on temp dirs (`crates/engine/tests/workspace.rs`) |
| Project view, VFS | 6 | 8% (est.) | views list (project / scopes), toolbar (Select Opened File, Expand/Collapse All), sort and display options, show/hide excluded, distinct icons for library and excluded folders, file colours, speed search, preview, file operations | ~37 (est.) | `lutra-vfs` (M1.2): lazy sorted listing, ignore rules (.gitignore, .ignore, info/exclude, global excludes, Lutra defaults), native file watching via `notify` with a polling fallback, capped UTF-8 reads, atomic writes; 34 tests on temp dirs, watcher tests run on both backends (`crates/vfs/tests`). M1.5: project tree UI (lazy listing, keyboard navigation, dimmed ignored entries, event refresh) in `ui/src/components/project/`, Vitest against a mocked engine. M1.3: `project.open`, `project.close`, `project.children` (watches listed directories, capped at 256 native / 32 polling, least recently listed dropped), `treeChanged` events from disk, `events.subscribe` on the control channel with a TCP test (`crates/control/tests/events.rs`). M1.4: the tree runs against the engine in the app: folder picker (`project.openDialog`) and `lutra-desktop --control PORT PATH` open a project, the tree lists it lazily with ignored (`var/` from `.gitignore`) and excluded (`node_modules`) entries dimmed, a file created on disk appears in an expanded folder within ~100 ms; verified with `ui.screenshot` on Windows 11 and Vitest (`ui/tests/project/`, `ui/tests/ipcEvents.test.ts`). M1.13: a directory listed only partly (more than 100,000 entries) ends with a "Listing truncated" note row (Vitest). M1.12: a folder that cannot be opened from the picker or the command line (including `unsaved_changes` when switching projects) and stopped file watching show as toasts instead of stderr (`apps/lutra-desktop/src/host.rs`, `ui/src/stores/notifications.ts`, `ui/tests/notifications/`, 19 Vitest tests; live check pending, gaming mode); score unchanged. Graded against `plan/phpstorm/02-project-view.md`: tool window, folder-first tree, dimmed ignored/excluded, keyboard navigation and live refresh exist; most toolbar, view and display options do not |
| Navigation | 10 | 0% | everything (M1.8: a command palette over commands only, no go-to-file/class/symbol yet; score unchanged) | ~120 (est.) | M1.8: `ui/src/components/palette/`, `ui/src/commands/` (fuzzy ranking, recent commands, global keymap), Vitest `ui/tests/palette/`, `ui/tests/commands/` |
| Navigation | 10 | 0% | everything (M1.8: a command palette over commands only, no go-to-file/class/symbol yet; score unchanged) | ~120 (est.) | M2.1: file outline in `crates/syntax` (namespaces braced and unbraced, classes, interfaces, traits, enums with cases, functions, methods, properties incl. promoted ones, constants; UTF-16 ranges and short signatures), the base for the Structure window (M2.7) and the symbol index (M2.5); not reachable from the UI yet, score unchanged. M1.8: `ui/src/components/palette/`, `ui/src/commands/` (fuzzy ranking, recent commands, global keymap), Vitest `ui/tests/palette/`, `ui/tests/commands/` |
| Search and replace | 6 | 0% | everything | ~80 (est.) | — |
| Code completion | 10 | 0% | everything | ~250 (est.) | — |
| Inspections and quick-fixes | 8 | 0% | everything | ~300 (est.) | — |
@ -63,6 +63,7 @@ The project tree is the first user-visible feature (Project view 8%, estimated);
| Date | Change | Summary |
|---|---|---|
| 2026-10-16 | minor | M2.1: Editor core and Navigation evidence (tree-sitter parse, outline, folding library), scores unchanged |
| 2026-10-16 | minor | M1.9b: Settings row 4% → 7% (est.), settings dialog; M1 done |
| 2026-10-16 | minor | M1.8: Navigation and Settings/keymaps evidence (palette, global keymap), scores unchanged |
| 2026-10-16 | minor | M1.12: Project view evidence (picker errors as toasts), scores unchanged |