M2.3: find in files backend
crates/search: parallel ignore walk + grep-searcher with the project's
ignore rules (vendor searched as library, node_modules and .git skipped),
literal or regex, case, whole word, include/exclude globs, binary and
> 8 MiB skips, match cap, cancel token, buffer overlay.
Engine: search.find returns {searchId} at once and runs off the lock;
pump_vfs drains results into searchResults (one per file, at most 500
per pump) and searchDone; search.cancel. Testkit: generated search tree.
Tests: library unit and property tests, engine tests, TCP control test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
a6b2fd1300
commit
172693c8dd
21 changed files with 2231 additions and 5 deletions
|
|
@ -2,3 +2,234 @@
|
|||
//!
|
||||
//! Streams matches from the project files off the UI thread, honouring the
|
||||
//! ignore rules of the project model.
|
||||
//!
|
||||
//! # Query
|
||||
//! A [`Query`] is compiled once into a [`Search`] ([`Search::new`]): a
|
||||
//! literal or a regular expression (Rust `regex` syntax), case sensitive or
|
||||
//! not, optionally whole words only, plus include and exclude globs. An
|
||||
//! invalid pattern or glob fails with [`SearchError`] before anything runs.
|
||||
//! Patterns never match across lines.
|
||||
//!
|
||||
//! # Which files are searched
|
||||
//! The walk follows the rules of `lutra-vfs`:
|
||||
//! - `.gitignore`, `.ignore`, `.git/info/exclude` and the global git excludes
|
||||
//! apply whether or not the folder is a git repository; ignored files and
|
||||
//! everything below an ignored directory are skipped. Ignore files above
|
||||
//! the project root are not read.
|
||||
//! - `.git` (directory or file) at any depth and every `node_modules`
|
||||
//! directory are skipped.
|
||||
//! - `vendor/` directly under the root is library code and is searched even
|
||||
//! when an ignore file lists it (Composer projects usually do); ignore
|
||||
//! files inside it still apply.
|
||||
//! - Hidden (dot) files are searched, as the project tree shows them.
|
||||
//! Symbolic links are not followed.
|
||||
//! - Include globs keep only files that match one of them, exclude globs
|
||||
//! drop files and whole directories. A glob without `/` matches the file
|
||||
//! or directory name (`*.php`), a glob with `/` the project-relative path
|
||||
//! (`src/**/*.php`).
|
||||
//! - Files larger than [`MAX_FILE_SIZE`], binary files (a NUL byte) and
|
||||
//! files that cannot be read are skipped.
|
||||
//!
|
||||
//! # Results
|
||||
//! Each file with matches is reported once as [`FileMatches`], in no
|
||||
//! particular order (the walk is parallel). Positions are 0-based; columns
|
||||
//! and lengths count UTF-16 code units. Lines end at `\n` (a `\r` before it
|
||||
//! is not part of the line). The preview is the line, or a window of at most
|
||||
//! [`MAX_PREVIEW_CHARS`] characters around the match for longer lines.
|
||||
//! At most [`Query::max_results`] matches are reported in total; the
|
||||
//! [`Summary`] says whether the cap cut the search short.
|
||||
//!
|
||||
//! # Threads and cancellation
|
||||
//! [`run`] blocks the calling thread and searches on a pool of up to
|
||||
//! [`MAX_THREADS`] walker threads; [`spawn`] runs it on a thread of its own
|
||||
//! and delivers [`SearchEvent`]s over a channel. A [`CancelToken`] stops a
|
||||
//! running search between files and between lines.
|
||||
//!
|
||||
//! # Open buffers
|
||||
//! An [`Overlay`] maps project-relative paths to text that replaces the file
|
||||
//! on disk, so unsaved edits are searched instead of the stale file. Only
|
||||
//! files the walk visits are looked up; text for a path that is not on disk
|
||||
//! (or is skipped by the rules) is not searched.
|
||||
|
||||
mod lines;
|
||||
mod query;
|
||||
mod walk;
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::mpsc::{Receiver, channel};
|
||||
use std::time::Duration;
|
||||
|
||||
pub use query::Search;
|
||||
pub use walk::run;
|
||||
|
||||
/// Largest file searched, in bytes (8 MiB).
|
||||
pub const MAX_FILE_SIZE: u64 = 8 * 1024 * 1024;
|
||||
|
||||
/// Default cap on the number of matches of one search.
|
||||
pub const DEFAULT_MAX_RESULTS: usize = 10_000;
|
||||
|
||||
/// Highest cap a query may ask for.
|
||||
pub const MAX_RESULTS_LIMIT: usize = 100_000;
|
||||
|
||||
/// Longest preview, in characters (Unicode scalar values).
|
||||
pub const MAX_PREVIEW_CHARS: usize = 300;
|
||||
|
||||
/// Most walker threads of one search.
|
||||
pub const MAX_THREADS: usize = 8;
|
||||
|
||||
/// What to search for and where.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Query {
|
||||
/// The text or regular expression; must not be empty.
|
||||
pub pattern: String,
|
||||
/// Treat `pattern` as a regular expression instead of literal text.
|
||||
pub regex: bool,
|
||||
/// Match letter case exactly.
|
||||
pub case_sensitive: bool,
|
||||
/// Only matches that start and end at word boundaries.
|
||||
pub whole_word: bool,
|
||||
/// Only files matching one of these globs; all files when empty.
|
||||
pub include: Vec<String>,
|
||||
/// Skip files and directories matching one of these globs.
|
||||
pub exclude: Vec<String>,
|
||||
/// Stop after this many matches; clamped to `1..=`[`MAX_RESULTS_LIMIT`].
|
||||
pub max_results: usize,
|
||||
}
|
||||
|
||||
impl Query {
|
||||
/// A literal, case-insensitive query for `pattern` with the default cap.
|
||||
pub fn literal(pattern: impl Into<String>) -> Self {
|
||||
Self {
|
||||
pattern: pattern.into(),
|
||||
regex: false,
|
||||
case_sensitive: false,
|
||||
whole_word: false,
|
||||
include: Vec::new(),
|
||||
exclude: Vec::new(),
|
||||
max_results: DEFAULT_MAX_RESULTS,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Why a query cannot run.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
|
||||
pub enum SearchError {
|
||||
/// The pattern is empty.
|
||||
#[error("the search text is empty")]
|
||||
EmptyPattern,
|
||||
/// The pattern is not a valid regular expression, or matches a line
|
||||
/// break.
|
||||
#[error("invalid pattern: {0}")]
|
||||
InvalidPattern(String),
|
||||
/// An include or exclude glob is invalid.
|
||||
#[error("invalid glob: {0}")]
|
||||
InvalidGlob(String),
|
||||
}
|
||||
|
||||
/// One match.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Match {
|
||||
/// 0-based line.
|
||||
pub line: usize,
|
||||
/// Start in the line, in UTF-16 code units.
|
||||
pub column: usize,
|
||||
/// Length in UTF-16 code units; never 0.
|
||||
pub length: usize,
|
||||
/// The line, or a window of it around the match, without the line break.
|
||||
pub preview: String,
|
||||
/// Start of the match inside `preview`, in UTF-16 code units.
|
||||
pub preview_column: usize,
|
||||
}
|
||||
|
||||
/// The matches in one file, in line order.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct FileMatches {
|
||||
/// Project-relative path with `/` separators.
|
||||
pub path: String,
|
||||
/// The matches; never empty.
|
||||
pub matches: Vec<Match>,
|
||||
}
|
||||
|
||||
/// How a search ended.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub struct Summary {
|
||||
/// Files with at least one reported match.
|
||||
pub files: usize,
|
||||
/// Matches reported.
|
||||
pub matches: usize,
|
||||
/// The match cap stopped the search; more matches may exist.
|
||||
pub truncated: bool,
|
||||
/// The search was cancelled before it finished.
|
||||
pub cancelled: bool,
|
||||
/// Wall time from start to end.
|
||||
pub elapsed: Duration,
|
||||
}
|
||||
|
||||
/// What a search started with [`spawn`] delivers, in order: any number of
|
||||
/// files, then exactly one summary.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum SearchEvent {
|
||||
/// A file with matches.
|
||||
File(FileMatches),
|
||||
/// The search ended; nothing follows.
|
||||
Done(Summary),
|
||||
}
|
||||
|
||||
/// Text of open buffers by project-relative path (`/` separators), searched
|
||||
/// instead of the file on disk.
|
||||
pub type Overlay = HashMap<String, String>;
|
||||
|
||||
/// Stops a running search. Clones share the flag.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct CancelToken(Arc<AtomicBool>);
|
||||
|
||||
impl CancelToken {
|
||||
/// A token that is not cancelled.
|
||||
pub fn new() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
/// Asks the search to stop; it ends with [`Summary::cancelled`] set.
|
||||
pub fn cancel(&self) {
|
||||
self.0.store(true, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Whether [`CancelToken::cancel`] was called.
|
||||
pub fn is_cancelled(&self) -> bool {
|
||||
self.0.load(Ordering::Relaxed)
|
||||
}
|
||||
}
|
||||
|
||||
/// Runs [`run`] on a new thread and returns the channel of its events. A
|
||||
/// receiver that is dropped does not stop the search; cancel it with
|
||||
/// `cancel`.
|
||||
///
|
||||
/// # Parameters
|
||||
/// - `root`: the project root (absolute).
|
||||
/// - `search`: the compiled query.
|
||||
/// - `overlay`: buffer texts that replace files on disk.
|
||||
/// - `cancel`: stops the search.
|
||||
///
|
||||
/// # Errors
|
||||
/// The OS refused to start the thread.
|
||||
pub fn spawn(
|
||||
root: PathBuf,
|
||||
search: Search,
|
||||
overlay: Overlay,
|
||||
cancel: CancelToken,
|
||||
) -> std::io::Result<Receiver<SearchEvent>> {
|
||||
let (tx, rx) = channel();
|
||||
std::thread::Builder::new()
|
||||
.name("lutra-search".into())
|
||||
.spawn(move || {
|
||||
let files = tx.clone();
|
||||
let summary = run(&root, &search, &overlay, &cancel, &move |file| {
|
||||
let _receiver_gone = files.send(SearchEvent::File(file));
|
||||
});
|
||||
let _receiver_gone = tx.send(SearchEvent::Done(summary));
|
||||
})?;
|
||||
Ok(rx)
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue