M2.5: symbol index and go-to commands
crates/index: walks the project with the search/VFS rules (vendor as library), parses .php files from tree-sitter outlines on a worker pool off the engine lock, updates incrementally from disk and buffers through an updater thread, caps at 200,000 files, and answers fuzzy file, class and symbol queries (nucleo-matcher, MPL-2.0 as a separate crate). Engine: index.status, goto.file, goto.class, goto.symbol; events indexProgress and indexReady drained by the pump; the index starts on project.open and is cancelled on close. Tests: unit, incremental, property (random queries and sources), engine and TCP control tests, release timing on a generated 20k-file project (3.9-4.2 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
f13a49064d
commit
053441d066
26 changed files with 3455 additions and 9 deletions
|
|
@ -1,4 +1,145 @@
|
|||
//! Symbol index for Lutra (layer L2).
|
||||
//!
|
||||
//! Stores declarations and usages for go-to and find-usages, starting with
|
||||
//! symbols extracted from tree-sitter trees.
|
||||
//! Lists every file of a project and the declarations of its PHP files
|
||||
//! (classes, interfaces, traits, enums, functions, methods, properties,
|
||||
//! constants, enum cases), extracted from tree-sitter outlines
|
||||
//! ([`lutra_syntax::outline`]), and answers fuzzy go-to queries over them.
|
||||
//!
|
||||
//! # Which files
|
||||
//! The walk follows the rules of `lutra-vfs` and `lutra-search`:
|
||||
//! `.gitignore`, `.ignore`, `.git/info/exclude` and the global git excludes
|
||||
//! apply without a git repository; `.git` and every `node_modules` directory
|
||||
//! are skipped; the root's `vendor/` is walked even when an ignore file
|
||||
//! lists it and its files are marked as library code; dot files are listed;
|
||||
//! symbolic links are not followed. Every file is listed for
|
||||
//! [`Index::find_files`]; only `.php` files are parsed, and only those up to
|
||||
//! [`MAX_PARSE_BYTES`]. At most [`MAX_FILES`] files are indexed.
|
||||
//!
|
||||
//! # Building
|
||||
//! [`build`] walks the project, then parses the PHP files on a pool of
|
||||
//! worker threads (one parser each, one file in memory each), project files
|
||||
//! before library files and larger files first within each group. It
|
||||
//! reports [`Update`]s through a callback: batches of indexed files,
|
||||
//! progress, and a final [`Summary`]. [`spawn_build`] runs it on a thread of
|
||||
//! its own and sends the updates over a channel. [`Updater`] re-indexes
|
||||
//! single files and directories later, from disk or from an editor buffer.
|
||||
//! A [`CancelToken`] stops both between files.
|
||||
//!
|
||||
//! # Robustness
|
||||
//! Source files are untrusted: reads are capped, invalid UTF-8 is replaced,
|
||||
//! broken PHP yields the declarations that could be recognised, symbols per
|
||||
//! file are capped ([`MAX_SYMBOLS_PER_FILE`]), and a panic while parsing one
|
||||
//! file (a bug) loses that file's symbols, not the index.
|
||||
|
||||
mod build;
|
||||
mod query;
|
||||
mod symbol;
|
||||
mod updater;
|
||||
mod walk;
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::Duration;
|
||||
|
||||
pub use build::{build, spawn_build};
|
||||
pub use query::{FileHit, Index, MAX_LIMIT, MAX_QUERY_CHARS, SymbolHit};
|
||||
pub use symbol::{IndexedFile, Symbol, SymbolKind, index_text, is_library, symbols_from_outline};
|
||||
pub use updater::{Job, Updater};
|
||||
|
||||
/// Most files one index holds.
|
||||
pub const MAX_FILES: usize = 200_000;
|
||||
|
||||
/// Largest PHP file parsed, in bytes (2 MiB). Larger files are listed for
|
||||
/// [`Index::find_files`] without symbols: tree-sitter-php parses about
|
||||
/// 2–3 MB/s, so one such file would hold a worker for a second or more, and
|
||||
/// files this size are generated code rather than declarations to navigate.
|
||||
pub const MAX_PARSE_BYTES: u64 = 2 * 1024 * 1024;
|
||||
|
||||
/// Most symbols kept from one file; the rest are dropped.
|
||||
pub const MAX_SYMBOLS_PER_FILE: usize = 20_000;
|
||||
|
||||
/// Most worker threads of one build.
|
||||
pub const MAX_THREADS: usize = 32;
|
||||
|
||||
/// Files per [`Update::Files`] batch of a build.
|
||||
pub const BATCH_FILES: usize = 256;
|
||||
|
||||
/// What a build or an [`Updater`] reports.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum Update {
|
||||
/// Indexed files that replace any earlier entry with the same path.
|
||||
Files {
|
||||
/// The files.
|
||||
files: Vec<IndexedFile>,
|
||||
/// From an [`Updater`] (a change after the build started), so newer
|
||||
/// than anything the build reports for the same path.
|
||||
live: bool,
|
||||
},
|
||||
/// A file or directory is gone: drop `path` and everything below it.
|
||||
Removed {
|
||||
/// Project-relative path with `/` separators.
|
||||
path: String,
|
||||
},
|
||||
/// Build progress: `done` of `total` files are indexed.
|
||||
Progress {
|
||||
/// Files indexed so far.
|
||||
done: usize,
|
||||
/// Files found by the walk.
|
||||
total: usize,
|
||||
},
|
||||
/// The build ended; nothing from it follows.
|
||||
Ready(Summary),
|
||||
}
|
||||
|
||||
/// How a build ended.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub struct Summary {
|
||||
/// Files indexed.
|
||||
pub files: usize,
|
||||
/// Symbols found in them.
|
||||
pub symbols: usize,
|
||||
/// PHP files parsed.
|
||||
pub parsed: usize,
|
||||
/// Bytes of PHP parsed.
|
||||
pub parsed_bytes: u64,
|
||||
/// The walk found more than [`MAX_FILES`] files; the rest are left out.
|
||||
pub capped: bool,
|
||||
/// The build was cancelled before it finished.
|
||||
pub cancelled: bool,
|
||||
/// Wall time of the walk alone.
|
||||
pub walk: Duration,
|
||||
/// Wall time from start to end.
|
||||
pub elapsed: Duration,
|
||||
}
|
||||
|
||||
/// Stops a running build or updater. Clones share the flag.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct CancelToken(Arc<AtomicBool>);
|
||||
|
||||
impl CancelToken {
|
||||
/// A token that is not cancelled.
|
||||
pub fn new() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
/// Asks the work to stop at the next file.
|
||||
pub fn cancel(&self) {
|
||||
self.0.store(true, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Whether [`CancelToken::cancel`] was called.
|
||||
pub fn is_cancelled(&self) -> bool {
|
||||
self.0.load(Ordering::Relaxed)
|
||||
}
|
||||
}
|
||||
|
||||
/// Worker threads for a build on this machine: one less than the cores,
|
||||
/// so the UI and the engine keep a core, at least one and at most
|
||||
/// [`MAX_THREADS`].
|
||||
pub fn default_threads() -> usize {
|
||||
std::thread::available_parallelism()
|
||||
.map(|n| n.get())
|
||||
.unwrap_or(2)
|
||||
.saturating_sub(1)
|
||||
.clamp(1, MAX_THREADS)
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue