Plugin system

Production CLI import uses source/file adapters followed by API v2 ingest, message and content pipelines. Python plugins run through the interface in the host interpreter; the Rust PST importer is an external executable. The diagram shows those pipelines and their GUI controls.

Open the graphic for a larger view. The file-parser layer is a subtree of local container acquisition, which supplies the three pipelines below. Planned geography and visualization extension points are not implemented plugin trees.

Processing and handoffs

Source discovery and file-format plugins recover original messages before the three processing pipelines:

1. Ingest: ClamAV scans the raw message at rank 1. Infected messages go through the write service to INFECTED and stop. At rank 2, the filing/handoff plugin deduplicates, publishes the original bytes, and durably queues message processing.
2. Message processing: header extraction stores basic metadata in SQLite. A handoff plugin queues content processing; other per-message subscribers can register independently.
3. Content processing: the MIME extractor emits typed body and attachment objects for subscribers. Identity extraction is the first evidence consumer; authoritative person matching remains later work.

The framework completes each rank before applying higher-ranked subscribers to the same type. Required cross-type outputs precede their consumers. A plugin may stop the current part, stop the message, or fail the import. Every plugin has access to all headers and all content of its current message. Types and body/attachment settings control invocation, not access. The filing plugin may read whatever it needs after ClamAV; full metadata storage stays in the second pipeline.

The TOML settings declare rank, body/attachment/both scope, and scanner timeout (60 seconds by default). End-of-run statistics report each plugin's invocation count, total time, shortest, longest, and average invocation. About lists registered processors by subscribed type, pipeline, rank, scope and timeout.

API v2 processors can call item.get_my_config() for their own configuration dictionary, with archive settings overriding installation settings recursively. item.write_my_config(values, scope="archive") replaces their archive layer; scope="installation" updates defaults shared across archives. The stable plugin registration name selects its namespace. Writes are staged until their rank succeeds, and unrelated settings are preserved. See the configuration API for layer reads, paths, conflict handling and failure semantics.

Incomplete work

Opening an archive with unfinished work shows Incomplete work with two initially checked choices: Continue ingest and Continue content processing. Confirmed work resumes from durable checkpoints. Unchecking both opens the archive without starting either job. The dialog returns on the next opening while work remains. When both are selected, ingest finishes before deferred content processing. Selecting only content processing handles eligible already-ingested messages, even while other ingest work remains unfinished. Required message-metadata jobs precede their content jobs.

Synthetic content and attached messages

Synthetic or virtual objects inherit their source's body/attachment scope and provenance. If the message body has no plain-text alternative, prefer HTML over RTF when creating synthetic text/plain content. A text attachment does not count as a plain-text body. Synthetic representations are processing inputs, not new canonical mail or MIME parts added to the original message.

Attached emails are first-class child messages with durable paths back to their parent message and MIME part. The viewer displays those paths and the attachment tag. They start independent jobs at message processing, bypassing ingest and another antivirus scan. The child retains its parent's scan provenance. The handoff uses shared deduplication/publication services to establish the child record and content reference. This creates a new job rather than a cycle within one job's DAG. Their own bodies and attachments have local scopes; the child message retains its attachment-origin provenance. Depth and expansion limits must prevent unbounded recursion while preserving the parent bytes.

Standard Message-ID and raw SHA-256 deduplication shares one canonical message and extracted metadata across standalone and attached copies. Each occurrence retains its own provenance, including the parent and MIME-part path. An attachment tag means the message was also observed as an attachment.

Child messages receive an attachment tag whose initial background is 5% gray. The planned tag editor will store tag IDs, names, nullable background/text colors, crosshatching, and font choices in SQLite, with configurable tagged-message display.

See the full plugin contract for the implemented API and the proposed extension.

Writing a processor

The processor contract specifies the API v2 manifest, one typed processing object and result, framework content types, rank barriers, aborts, timeout enforcement, archive services and durable handoffs. Existing source/file API v1 plugins remain supported before raw-message dispatch. Use make run ARGS="--archive PATH processors" to inspect the production registry, processing-status to inspect work, and process --phase content to resume content without source files. identities addresses and identities organizations expose picker data through the CLI. Plugins use cooperative cancellation and I/O deadlines; late results are rejected before publication.

The complete design checkpoints plugin versions, configuration and input identity so retries can skip completed work. Publication and queue handoff share a recovery protocol. Scanner failure never counts as clean; child messages retain parent scan provenance. Manual identity edits and tags survive processing reruns.