Local Text SearchLocal corpus search with background jobs

The problem & workflow

Authenticated user creates a search → worker scans configured synthetic corpus → job progress and output file are updated → owner browses/downloads results → administrator reviews usage.

Key features

• Memory-mapped text scanning and Python fallback
• Optional Cython acceleration
• Background job progress and batched result-file writes
• Result pagination/downloads and owner checks
• User/admin management and usage statistics
• Optional Telegram ingestion, disabled by default

Architecture & decisions

• Corpus, database and output directories resolve through one module instead of the old machine-specific Downloads path. Authentication and web handlers must use the same database.
• mmap avoids loading each entire file into a Python string. Case-insensitive matching currently lowers byte strings; this is not general Unicode case folding.
• Windows uses the threaded worker path; other platforms retain multiprocessing-related code. Describe measured behavior on the tested OS.
• Optional Telegram and acceleration packages are split from the minimal Flask install. Enabling one integration should be deliberate.
• Session secrets can persist through an environment value; generating a new key at restart invalidates existing sessions.

Inside the source

• config.py
• app.py
• auth.py
• search_engine.py
• database.py
• search_engine_cython.pyx

Keeping long-running corpus search outside a web request

I built this local search application with Flask, background jobs and memory-mapped file scanning.

Local Text Search uses Flask for the user and administration experience while background jobs scan memory-mapped files. SQLite stores job metadata, batched result files hold matches, and the interface polls progress before showing paginated results or a download. Result ownership is checked at the application boundary.

Worker behavior depends on the platform. The documented Windows path uses threading, and the optional Cython matcher is a separate build path. One configuration source now controls corpus, database, result and session locations, so a configurable corpus no longer conflicts with a hard-coded search-engine default.

A result limit must be shared across workers

Giving each file worker its own limit can return more results than a user requested. The release uses a shared result budget and checks matching across long lines, mixed bytes and multiple files. Six regression/access tests cover those cases and cross-user boundaries; interrupted jobs and acceleration equivalence remain further verification tasks.

The benchmark uses 800,000 generated lines and documents bytes, matches, Windows hardware, Python version and three measured times. OS caches were not flushed and Cython was not enabled. These details make the measurement reproducible without presenting an unmeasured speedup or a claim about terabyte-scale production data.

Project status

Local corpus-processing and administration application. Public demos use generated text only.

Use Python 3.12 for the recorded checks. Export .env.example values into the process; config.py resolves relative paths from the application root. Set a persistent SECRET_KEY for sessions and ENABLE_TELEGRAM=0. bootstrap_demo.py prompts for a fresh 12-character administrator password and writes generated text only when absent. Cython is optional and requires a compiler; Python remains the default verified path.