M2.5: symbol index and go-to commands

crates/index: walks the project with the search/VFS rules (vendor as
library), parses .php files from tree-sitter outlines on a worker pool
off the engine lock, updates incrementally from disk and buffers through
an updater thread, caps at 200,000 files, and answers fuzzy file, class
and symbol queries (nucleo-matcher, MPL-2.0 as a separate crate).

Engine: index.status, goto.file, goto.class, goto.symbol; events
indexProgress and indexReady drained by the pump; the index starts on
project.open and is cancelled on close. Tests: unit, incremental,
property (random queries and sources), engine and TCP control tests,
release timing on a generated 20k-file project (3.9-4.2 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
danny 2026-10-17 12:53:33 +02:00
parent f13a49064d
commit 053441d066
26 changed files with 3455 additions and 9 deletions

View file

@ -0,0 +1,511 @@
//! The symbol index and the go-to commands: `index.status`, `goto.file`,
//! `goto.class` and `goto.symbol`.
//!
//! # Index lifecycle
//! [`sync`] runs on every [`crate::Engine::pump_vfs`] and before every
//! index command. When a project is open and no index exists for its root,
//! it starts one: [`lutra_index::spawn_build`] walks and parses the project
//! on background threads, and an [`Updater`] thread handles later changes.
//! Both send [`Update`]s over one channel; [`sync`] merges them into the
//! in-memory [`Index`] in [`State`], so the engine lock is only held for
//! cheap inserts, never while files are walked or parsed. Progress becomes
//! [`Event::IndexProgress`] (at most every [`PROGRESS_EVERY`]) and the end
//! of the build [`Event::IndexReady`]. Closing the project (or opening
//! another one) drops the index and cancels its threads.
//!
//! # Incremental updates
//! - Disk: [`note_disk_changes`] turns the workspace's VFS events into
//! updater jobs (created or modified files are re-read, created or
//! renamed directories rescanned) and drops removed paths at once. The
//! project watcher is not recursive (see [`crate::workspace`]), so changes
//! in directories that no client listed and no buffer pins are not seen
//! until the next project open.
//! - Buffers: when an open buffer's version changes, its text is
//! re-indexed instead of the file on disk; when it is closed with unsaved
//! changes, the file on disk is re-read. Only files the index already
//! lists (or, while building, may still list) are re-indexed from
//! buffers, so a buffer of an ignored file never enters the index.
//! - Updates from the updater win over build results for the same path
//! that arrive later.
//!
//! # Queries
//! Queries run on the current index, never wait for the build, and report
//! `indexing: true` while it runs. See [`lutra_index::Index`] for matching
//! and ranking.
use std::collections::{BTreeMap, HashSet};
use std::path::PathBuf;
use std::sync::mpsc::{Receiver, TryRecvError, channel};
use std::time::{Duration, Instant};
use lutra_index::{CancelToken, Index, Job, SymbolHit, Update, Updater};
use lutra_vfs::{EntryKind, VfsEvent};
use serde::Deserialize;
use serde_json::{Value, json};
use crate::buffers::BufferId;
use crate::cmd::params::parse;
use crate::workspace::path_string;
use crate::{Command, CommandSpec, EngineError, Event, Outcome, State};
/// Shortest time between two [`Event::IndexProgress`].
pub const PROGRESS_EVERY: Duration = Duration::from_millis(250);
/// Results of a go-to query when the params give no `limit`.
pub const DEFAULT_LIMIT: usize = 50;
/// Most updates merged by one [`sync`]; the rest wait for the next one, so
/// a pump never holds the engine lock for long.
pub const MAX_UPDATES_PER_SYNC: usize = 4_096;
/// The index of the open project, if one is being built or ready.
#[derive(Debug, Default)]
pub struct IndexState {
/// The index and its background threads.
running: Option<Running>,
}
impl IndexState {
/// Whether the initial build is still running.
pub fn indexing(&self) -> bool {
self.running.as_ref().is_some_and(|r| r.building)
}
/// The index, if a project is open.
pub fn index(&self) -> Option<&Index> {
self.running.as_ref().map(|r| &r.index)
}
}
/// The index of one project root and its background work.
#[derive(Debug)]
struct Running {
/// The project root.
root: PathBuf,
/// The files and symbols known so far.
index: Index,
/// Stops the build and the updater.
cancel: CancelToken,
/// Updates from the build and the updater.
updates: Receiver<Update>,
/// Re-indexes changed files; `None` if its thread could not start.
updater: Option<Updater>,
/// The initial build has not ended yet.
building: bool,
/// When the build started.
started: Instant,
/// The latest progress not yet emitted.
progress: Option<(usize, usize)>,
/// When progress was last emitted.
progress_at: Option<Instant>,
/// Paths the updater reported while the build runs; later build results
/// for them are stale.
touched: HashSet<String>,
/// The path and version each open buffer was last seen at.
buffers: BTreeMap<BufferId, (String, u64)>,
/// Paths whose index entry comes from an unsaved buffer, so disk
/// changes don't replace it.
overlay: HashSet<String>,
}
impl Drop for Running {
/// Cancels the build and the updater; their threads end at the next
/// file.
fn drop(&mut self) {
self.cancel.cancel();
}
}
impl Running {
/// Starts indexing `root`. Threads that cannot start leave an index that
/// stays empty (build) or is not updated (updater); queries still work.
fn start(root: PathBuf) -> Self {
let (tx, updates) = channel();
let cancel = CancelToken::new();
let building = lutra_index::spawn_build(
root.clone(),
lutra_index::default_threads(),
cancel.clone(),
tx.clone(),
)
.is_ok();
let updater = Updater::spawn(root.clone(), cancel.clone(), tx).ok();
Self {
root,
index: Index::new(),
cancel,
updates,
updater,
building,
started: Instant::now(),
progress: None,
progress_at: None,
touched: HashSet::new(),
buffers: BTreeMap::new(),
overlay: HashSet::new(),
}
}
/// Queues `job` for the updater.
fn send(&self, job: Job) {
if let Some(updater) = &self.updater {
updater.send(job);
}
}
/// Merges `update` into the index; returns the event it causes.
fn apply(&mut self, update: Update) -> Option<Event> {
match update {
Update::Files { files, live } => {
for file in files {
if live {
if self.building {
self.touched.insert(file.path.clone());
}
} else if self.touched.contains(&file.path) {
continue;
}
self.index.insert(file);
}
None
}
Update::Removed { path } => {
self.index.remove(&path);
if self.building {
self.touched.insert(path);
}
None
}
Update::Progress { done, total } => {
self.progress = Some((done, total));
None
}
Update::Ready(summary) => Some(self.ready(summary.elapsed)),
}
}
/// Ends the build after `elapsed` and returns [`Event::IndexReady`].
fn ready(&mut self, elapsed: Duration) -> Event {
self.building = false;
self.progress = None;
self.touched.clear();
Event::IndexReady {
files: self.index.len(),
symbols: self.index.symbol_count(),
elapsed_ms: u64::try_from(elapsed.as_millis()).unwrap_or(u64::MAX),
}
}
}
/// Brings the index in step with the state: starts it for a newly open
/// project, drops it without one, merges finished updates, re-indexes
/// changed buffers and emits progress and readiness events. Called by the
/// pump and by every index command.
pub fn sync(state: &mut State) {
let root = state
.project
.as_ref()
.map(|w| w.project.root().to_path_buf());
let Some(root) = root else {
state.index.running = None;
return;
};
if state.index.running.as_ref().is_none_or(|r| r.root != root) {
state.index.running = Some(Running::start(root));
}
let Some(running) = state.index.running.as_mut() else {
return;
};
let mut events = Vec::new();
for _ in 0..MAX_UPDATES_PER_SYNC {
match running.updates.try_recv() {
Ok(update) => events.extend(running.apply(update)),
Err(TryRecvError::Empty) => break,
Err(TryRecvError::Disconnected) => {
if running.building {
let elapsed = running.started.elapsed();
events.push(running.ready(elapsed));
}
break;
}
}
}
if let Some((done, total)) = running.progress
&& running.building
&& running
.progress_at
.is_none_or(|at| at.elapsed() >= PROGRESS_EVERY)
{
running.progress = None;
running.progress_at = Some(Instant::now());
events.push(Event::IndexProgress { done, total });
}
sync_buffers(state);
for event in events {
state.emit(event);
}
}
/// Re-indexes buffers whose text changed since the last sync, and re-reads
/// the files of closed buffers that were indexed from unsaved text.
fn sync_buffers(state: &mut State) {
let Some(running) = state.index.running.as_mut() else {
return;
};
let mut seen = BTreeMap::new();
for id in state.buffers.ids() {
let Some(open) = state.buffers.get(id) else {
continue;
};
let path = path_string(&open.path);
let dirty = open.dirty();
let changed = match running.buffers.get(&id) {
Some((old_path, version)) => *version != open.version || *old_path != path,
None => dirty,
};
let known = running.index.contains(&path) || running.building;
if changed && known {
if dirty {
running.overlay.insert(path.clone());
}
running.send(Job::File {
path: path.clone(),
text: Some(open.buffer.clone()),
});
}
if !dirty {
running.overlay.remove(&path);
}
seen.insert(id, (path, open.version));
}
let closed: Vec<String> = running
.buffers
.iter()
.filter(|(id, _)| !seen.contains_key(id))
.map(|(_, (path, _))| path.clone())
.collect();
for path in closed {
if running.overlay.remove(&path) {
running.send(Job::File { path, text: None });
}
}
running.buffers = seen;
}
/// Turns changes on disk into index updates: removed paths are dropped at
/// once, created and modified files are re-read, created directories and
/// rename targets are rescanned. Files indexed from an unsaved buffer keep
/// the buffer's text. Does nothing without an index.
pub fn note_disk_changes(state: &mut State, changes: &[VfsEvent]) {
let Some(running) = state.index.running.as_mut() else {
return;
};
for change in changes {
match change {
VfsEvent::Created { path, kind } => {
let path = path_string(path);
if *kind == EntryKind::Directory {
running.send(Job::Dir { path });
} else if !running.overlay.contains(&path) {
running.send(Job::File { path, text: None });
}
}
VfsEvent::Modified { path } => {
let path = path_string(path);
if !running.overlay.contains(&path) {
running.send(Job::File { path, text: None });
}
}
VfsEvent::Removed { path } => {
let path = path_string(path);
running.apply(Update::Removed { path });
}
VfsEvent::Renamed { from, to } => {
running.apply(Update::Removed {
path: path_string(from),
});
running.send(Job::Dir {
path: path_string(to),
});
}
VfsEvent::Failed(_) => {}
}
}
}
/// Drops the index and cancels its threads; `project.close` calls it.
pub fn close(state: &mut State) {
state.index.running = None;
}
/// `index.status`: whether the index is building and how big it is.
pub struct Status;
impl Command for Status {
fn spec(&self) -> &CommandSpec {
&CommandSpec {
id: "index.status",
label: "Index Status",
menu_path: None,
default_shortcut: None,
params: "none; returns { state: \"idle\" | \"indexing\", files, symbols }; events indexProgress { done, total } (at most every 250 ms) and indexReady { files, symbols, elapsedMs } report the build after project.open",
}
}
fn run(&self, state: &mut State, _params: Value) -> Result<Outcome, EngineError> {
sync(state);
let index = state.index.index();
Ok(Outcome::value(json!({
"state": if state.index.indexing() { "indexing" } else { "idle" },
"files": index.map_or(0, Index::len),
"symbols": index.map_or(0, Index::symbol_count),
})))
}
}
/// Params of the go-to commands.
#[derive(Deserialize)]
#[serde(deny_unknown_fields, rename_all = "camelCase")]
struct QueryParams {
/// What to look for.
query: String,
/// Most results.
limit: Option<usize>,
}
/// Parses the params of the go-to command `id`, brings the index up to
/// date and returns the query and the limit.
fn prepare(state: &mut State, id: &str, params: Value) -> Result<(String, usize), EngineError> {
let p: QueryParams = parse(id, params)?;
sync(state);
Ok((p.query, p.limit.unwrap_or(DEFAULT_LIMIT)))
}
/// The spec of a go-to command.
const fn goto_spec(id: &'static str, label: &'static str, params: &'static str) -> CommandSpec {
CommandSpec {
id,
label,
menu_path: None,
default_shortcut: None,
params,
}
}
/// `goto.file`: fuzzy search over the project's file paths.
pub struct File;
impl Command for File {
fn spec(&self) -> &CommandSpec {
const SPEC: CommandSpec = goto_spec(
"goto.file",
"Go to File",
"{ query, limit? (default 50, at most 1000) }; returns { items: [{ path, score, matches: [[start, end]] }], indexing }: fuzzy over project-relative paths, file names weighted higher, project files before vendor files; matches are UTF-16 ranges in path; indexing is true while the index is still being built (partial results)",
);
&SPEC
}
fn enabled(&self, state: &State) -> bool {
state.has_project()
}
fn run(&self, state: &mut State, params: Value) -> Result<Outcome, EngineError> {
let (query, limit) = prepare(state, self.spec().id, params)?;
let items: Vec<Value> = state
.index
.index()
.map(|index| index.find_files(&query, limit))
.unwrap_or_default()
.into_iter()
.map(|hit| json!({ "path": hit.path, "score": hit.score, "matches": hit.matches }))
.collect();
Ok(Outcome::value(
json!({ "items": items, "indexing": state.index.indexing() }),
))
}
}
/// `goto.class`: fuzzy search over classes, interfaces, traits and enums.
pub struct Class;
impl Command for Class {
fn spec(&self) -> &CommandSpec {
const SPEC: CommandSpec = goto_spec(
"goto.class",
"Go to Class",
"{ query, limit? (default 50, at most 1000) }; returns { items: [{ name, fqn, kind, path, line, column, library }], indexing }: classes, interfaces, traits and enums by name (by FQN when the query has a backslash), project before library; line and column (UTF-16) point at the name",
);
&SPEC
}
fn enabled(&self, state: &State) -> bool {
state.has_project()
}
fn run(&self, state: &mut State, params: Value) -> Result<Outcome, EngineError> {
let (query, limit) = prepare(state, self.spec().id, params)?;
let items: Vec<Value> = state
.index
.index()
.map(|index| index.find_classes(&query, limit))
.unwrap_or_default()
.iter()
.map(|hit| symbol_item(hit, false))
.collect();
Ok(Outcome::value(
json!({ "items": items, "indexing": state.index.indexing() }),
))
}
}
/// `goto.symbol`: fuzzy search over functions, methods, properties,
/// constants and enum cases.
pub struct Symbol;
impl Command for Symbol {
fn spec(&self) -> &CommandSpec {
const SPEC: CommandSpec = goto_spec(
"goto.symbol",
"Go to Symbol",
"{ query, limit? (default 50, at most 1000) }; returns { items: [{ name, fqn, kind, container, path, line, column, library }], indexing }: functions, methods, properties, constants and enum cases by name; Class::member matches the class and the member; container is the declaring class FQN or null",
);
&SPEC
}
fn enabled(&self, state: &State) -> bool {
state.has_project()
}
fn run(&self, state: &mut State, params: Value) -> Result<Outcome, EngineError> {
let (query, limit) = prepare(state, self.spec().id, params)?;
let items: Vec<Value> = state
.index
.index()
.map(|index| index.find_symbols(&query, limit))
.unwrap_or_default()
.iter()
.map(|hit| symbol_item(hit, true))
.collect();
Ok(Outcome::value(
json!({ "items": items, "indexing": state.index.indexing() }),
))
}
}
/// The JSON item of a class or symbol hit, with `container` for symbols.
fn symbol_item(hit: &SymbolHit<'_>, container: bool) -> Value {
let s = hit.symbol;
let mut item = json!({
"name": s.name,
"fqn": s.fqn,
"kind": s.kind.as_str(),
"path": hit.path,
"line": s.line,
"column": s.column,
"library": hit.library,
});
if container && let Some(map) = item.as_object_mut() {
map.insert("container".into(), json!(s.container));
}
item
}

View file

@ -3,6 +3,7 @@
mod app;
mod buffer;
mod file;
pub mod goto;
mod params;
mod project;
mod recovery;
@ -42,5 +43,9 @@ pub fn register_builtins(registry: &mut Registry) -> Result<(), EngineError> {
registry.register(Box::new(ui::RegisterCommands))?;
registry.register(Box::new(search::Find))?;
registry.register(Box::new(search::Cancel))?;
registry.register(Box::new(goto::Status))?;
registry.register(Box::new(goto::File))?;
registry.register(Box::new(goto::Class))?;
registry.register(Box::new(goto::Symbol))?;
Ok(())
}

View file

@ -58,6 +58,7 @@ pub fn close_project(state: &mut State, force: bool) -> Result<(), EngineError>
}
state.journal = None;
state.project = None;
crate::cmd::goto::close(state);
state.emit(Event::ProjectClosed);
Ok(())
}