MisterXIndexed text search and background processing

The problem & workflow

Authorized synthetic records are ingested → indexed search returns paginated results; the retained asynchronous path creates a job, scans its corpus, reports progress and returns owner-scoped output.

Key features

• Active OpenSearch search route
• Ingestion utilities and document indexing
• Pagination and search_after handling
• Optional Redis caching and invalidation
• Legacy file scanning, job polling/progress/results
• Subscriptions, account administration and usage limits

Architecture & decisions

• Indexed retrieval and direct file scanning are different execution paths. OpenSearch availability determines the active indexed route; keeping old job utilities does not make them automatic failover.
• Composer now declares the missing OpenSearch PHP client. Configure the index explicitly; the public default is `portfolio_records`.
• Legacy identifiers such as LEAKED_DATA_PATH remain in the source for compatibility and are documented, rather than presented as new terminology.
• A hard-coded personal unlimited-account exception is replaced with an empty-by-default environment allowlist.
• Educational intent does not authorize processing private information. The MIT warranty clause is not a promise of immunity.

Inside the source

• api/search/index.php
• api/utils/OpenSearchClient.php
• api/utils/opensearch_indexer.php
• api/utils/fast_search.php
• api/utils/process_search_job.php

Comparing indexed search with direct file processing

I built MisterX to explore indexed search, caching and background processing with account-based access.

MisterX retains multiple implementation paths: an active OpenSearch route and file-search/background-job code. These solve different parts of the search problem. Ingestion prepares indexed documents, search_after carries pagination state, and background processing provides a job lifecycle with progress and result ownership.

The release supplies generated JSONL records and a compatible Composer dependency setup. It does not publish private corpora, credential collections or acquisition instructions. Historical variable names remain documented so the code's evolution is understandable; an educational statement does not authorize processing somebody else's private data.

Query identity includes the index and cursor

A cached search response depends on more than its query text. Changing the index or pagination cursor changes the requested page, so those values belong in the cache key. I corrected that boundary and checks ownership before processing a job rather than after its work has begun.

PHP installation and synthetic database setup were verified locally. The supplied OpenSearch checks have not yet executed successfully here, so no indexed performance or equivalence benchmark is published. The next measured step is an isolated generated-corpus run covering pagination, service failure, cache behavior and agreement with the supported file path.

Project status

Search engineering showcase. Public examples use generated, authorized records only; no original corpus or private search results are included.

Use PHP 8.1+ with mysqli and optional Redis extension. Export env.example variables into PHP: disposable DB_* credentials, fresh JWT_SECRET, DEMO_MODE=1 and DEMO_PASSWORD. Start a dedicated OpenSearch instance; configure OPENSEARCH_HOST/PORT and OPENSEARCH_INDEX=portfolio_records. Redis is optional caching infrastructure. Set LEAKED_DATA_PATH only to generated authorized fixtures; the legacy variable name does not authorize private data. Ingestion uses fixtures/generated.jsonl.