Maintain an S3 Repository

Every runner sharing an S3 repository must use the same bucket and prefix and support its WAL3 admission protocol. Stop older binaries before upgrading because they do not observe the current holds. The S3 profile requires the s3 feature. Recovery APIs below are in casita::experimental.

Routine checks

casita holds s3://my-bucket/repository
casita holds s3://my-bucket/repository --json

casita holds reads collector ownership and the state and coordination pin ledgers without waiting for repository admission. It uses standard AWS credentials. Each ledger is a separate snapshot, so use exact tokens rather than matching entries by writer name or time. Inspection never releases a hold. A listed token does not prove its owner has stopped.

For shared root ownership, see Share an S3 Repository Across Owners.

Collection and reads

Mutation sessions and retained reads create durable pins. They protect staging data, selected payloads, snapshots, and metadata files while collection runs. Keep a retention hold until its payload stream closes. A WAL3 cursor alone does not protect physical payloads.

Only one collector runs at a time. collect waits for ownership; try_collect, try_collect_logical, and try_vacuum return Busy when another collector or a conflicting pin update prevents a pass. Holds do not expire. An abandoned collector token blocks later collect calls until it is recovered.

Collection first prunes unreachable logical records, then updates the payload catalog and deletes retired packs and manifests. An interrupted pass may leave physical garbage; vacuum reclaims deferred packs and obsolete catalog files. Rooted data and active pins remain protected.

Clean shutdown

For a library application, stop new work, await running operations, drop sessions and readers, then await casita::experimental::flush_repository_leases() before stopping Tokio. Dropping a session schedules its durable release; the final flush waits for those releases and background catalog work. A failed release reports an error and leaves the durable pin in place. The CLI drains releases on exit.

When a command is blocked

Run casita holds s3://BUCKET/PREFIX --json to see every token, pin scope, released history, prune fence, and deletion claim. Admission warnings show a short list of blocking tokens by default. For acquisition and release events, use --log-filter casita=info; casita=debug adds pin events.

Writer names are diagnostic. The CLI uses CASITA_WRITER when set, otherwise a process ID and random suffix. Even an explicit name can be reused. Correlate the exact token with process logs before deciding it is abandoned.

Recover an abandoned hold

Holds have no TTL. A paused process can still resume and read or publish data, so elapsed time is never sufficient evidence that its hold is abandoned.

1. Terminate the owning runner and prevent it from resuming. For an interrupted exclusive collector, stop every runner and allow outstanding backend requests to settle before recovery.
2. Run casita holds s3://BUCKET/PREFIX --json to inspect operational tokens and both pin ledgers. For interrupted payload collection, record the exact state.collector token as well as the operational collector token. The two tokens identify different ownership records.
3. Call release_abandoned_repository_hold(&hold.token) for the exact abandoned token. Reusing a diagnostic writer name does not grant ownership; replaying recovery for an old token cannot remove a newer token.
4. For interrupted payload collection, call Repository::recover_s3_collection(bucket, prefix, writer, &collector_token). It obtains collector ownership before opening payload discovery and can therefore reopen through an abandoned logical-prune fence. It preserves live pins and inherited deletion claims throughout marking and metadata commit, then finishes the sweep and catalog publication. For an already-open generic repository, use recover_collection(&collector_token).
5. Drain releases, check the remaining inventories, and run an integrity audit before resuming traffic. If a recovery attempt fails, inspect again: its new collector token may have replaced the abandoned token while preserving the original fence and claims. Never substitute an older token for the current one.

The operational collector token and state.collector are different records. Releasing the operational hold does not clear a prune fence or deletion claim.

For other abandoned records, first establish that the owner and its outstanding requests have stopped:

Record	Recovery API
Reader or writer pin	release(&exact_pin_token) on state.pin_store() or state.repository_coordination_pin_store(), according to its ledger
State WAL shard claims	state.recover_wal_deletions(exact_claim_tokens)
Coordination WAL claims	state.recover_repository_coordination_deletions(exact_claim_tokens)

WAL claim recovery needs the complete current token set for that ledger. It verifies that claimed paths are unreferenced and retains claims through retries. Pin release clears only the named pin. Reinspect after a failure before retrying because the current tokens may have changed.

Maintain the WALs

Wal3MetadataStore::collect_wal(reader_grace_period) collects obsolete state-log shards. collect_repository_coordination(reader_grace_period) compacts the separate admission log without expiring holds. Both acquire collector ownership and can run alongside ordinary readers and writers. Choose a grace period longer than the longest manifest-to-fragment metadata read. Pins protect files in use; unsettled deletion claims prevent path reuse until recovery. A failed collection retains its token for recovery.