hexasync.dataprotection

Envelope encryption (AES-256-GCM) with pluggable KEK custody and key rotation — the HexaSync at-rest data-protection library. Self-describing hxdp: wire format, code-registry algorithm versioning, per-service DEK keyring with in-memory cache, EF Core integration ([Encrypted]), and a self-healing DEK/KEK lifecycle driver.

The Azure Key Vault KEK adapter ships separately as hexasync.dataprotection.azurekeyvault.

How it works (30 seconds)

A per-service KEK (key-encryption key) wraps per-generation DEKs (data-encryption keys). Your data is encrypted with the active DEK; the wrapped DEKs live in a data_encryption_keys table in your database. On boot the library unwraps the keyring into an in-memory cache (one KEK round-trip), then encrypt/decrypt are local. It is hxdp:-only and fails closed — it never falls back to plaintext or a legacy format.

Install

dotnet add package hexasync.dataprotection
dotnet add package hexasync.dataprotection.azurekeyvault   # production KEK custody (optional in dev)

Configuration (flat DATA_PROTECTION_* env vars)

House convention: config via flat UPPER_SNAKE_CASE environment variables (no appsettings.json). Each maps to a property via [ConfigurationKeyName], bound as IOptions<DataProtectionConfig>.

DATA_PROTECTION_KEK_PROVIDER=Local              # "Local" (dev/low envs) or "AzureKeyVault" (prod); also the default KEK_ACTIVE
DATA_PROTECTION_KEK_ACTIVE=Local                # which registered provider WRAPS new/re-wrapped DEKs (default = KEK_PROVIDER).
                                                 # Register BOTH providers and flip this for a seamless Local↔Azure cutover
                                                 # (see "Multiple KEK providers" below). Unwrap always routes to whoever wrapped a DEK.
DATA_PROTECTION_MASTER_KEY_BASE64=<b64 32 bytes> # REQUIRED when Local is active/fallback (`openssl rand -base64 32`)
DATA_PROTECTION_MASTER_KEY_BASE64_OLD=          # the PREVIOUS master, set only while rotating it (see "Rotating the local master")
DATA_PROTECTION_KEK_ID=local                    # the id LEGACY rows carry. It no longer stamps new wraps — the kek_id is
                                                 # derived from the master itself (`local:<12 hex>`), so it tracks the key
DATA_PROTECTION_ACTIVE_SCHEME=v1                # optional; default = newest registered scheme; unknown → boot throws
DATA_PROTECTION_EXPECTED_MAX_PODS=16            # must be >= your HPA maxReplicas; drives the nonce early-rotation trip
DATA_PROTECTION_STARTUP_MAX_ATTEMPTS=5          # optional; bounded startup-warm retries (KEK may be transiently down)
DATA_PROTECTION_STARTUP_RETRY_DELAY=00:00:02    # optional

Local is for dev / low environments. The master key sits in config; a KV-wrapped DEK cannot be unwrapped locally (by design — prod data is not decryptable off-cluster). Production should use AzureKeyVault (see that package's README).

Key rotation has no env var. The DEK rotation period and the rest of the maintenance schedule are configured in code via the AddKeyringMaintenance(cfg => …) lambda — there is no DATA_PROTECTION_MAX_DEK_AGE and no DataProtection:KeyringMaintenance JSON section. See Key rotation.

Wiring (EF-backed, in order)

// 1. Your DbContext factory — the library needs a FRESH context per keyring call (the provider is a singleton;
//    a captured scoped DbContext would throw ObjectDisposedException under load).
services.AddDbContextFactory<MyDbContext>(o => o.UseNpgsql(connectionString));

// 2. Compose the library (store + provider + IDataProtector + startup warm); binds the DATA_PROTECTION_* env vars.
services.AddHexaSyncDataProtectionWithEf<MyDbContext>(configuration);   // bundles AddEfDekKeyringStore<MyDbContext>()

// 3. Azure Key Vault KEK custody. May be registered ALONGSIDE Local (both coexist — DATA_PROTECTION_KEK_ACTIVE picks
//    which one wraps; unwrap routes to whichever wrapped a given DEK). This is the seamless-cutover path (below).
services.AddAzureKeyVaultKeyWrapper(configuration);

// 4. Health check (readiness): after AddHexaSyncDataProtection.
services.AddHealthChecks().AddHexaSyncDataProtectionHealthCheck();

// 5. Opt-in self-healing maintenance (DEK rotation + KEK re-wrap). Requires a leader lock (below).
//    WITHOUT this call nothing ever rotates or re-wraps — see "Key rotation" for what that costs you.
services.AddSingleton<IDataProtectionLeaderLock, RedisMedallionLeaderLock>();
services.AddKeyringMaintenance(cfg =>
{
    cfg.LeaderKey = "hexasync:<your-service>:keyring-maintenance";   // per SERVICE — the default is shared
    cfg.MaxDekAge = TimeSpan.FromDays(180);                          // the DEK rotation period (this is the default)
});

A custom audit sink (optional) must be registered before AddHexaSyncDataProtection (it wins a TryAdd over the default no-op): services.AddSingleton<IDataProtectionAuditSink, MySink>();.

Leader-lock adapter (Medallion.Threading Redis)

AddKeyringMaintenance needs an IDataProtectionLeaderLock so exactly one replica per service runs the cycle. The library ships no Redis dependency — adapt your existing distributed lock (the shape is a single blocking acquire returning a handle held until disposed):

public sealed class RedisMedallionLeaderLock(IDistributedLockProvider locks) : IDataProtectionLeaderLock
{
    public async Task<IAsyncDisposable> AcquireAsync(string key, CancellationToken ct)
        => await locks.AcquireLockAsync(key, cancellationToken: ct);   // Medallion handle is already IAsyncDisposable
}

AcquireAsync should block until acquired (so a dead leader's lock is picked up as soon as it releases).

Key rotation (DEK age) and the maintenance cycle

The first DEK is created at startup regardless — but nothing rotates until you call AddKeyringMaintenance. AddHexaSyncDataProtection deliberately does not wire it (that would force an IDataProtectionLeaderLock on every consumer). Skip it and you get a DEK that encrypts forever, no KEK re-wrap when the KEK version changes, and no overdue signal — the health check's Degraded branch only runs when KeyringMaintenanceConfig is registered.

One elected leader per service runs a cycle every Interval, and each cycle: ensures an active DEK exists → rotates it if it is older than MaxDekAge (the previous DEK is retained so old data still decrypts) → re-wraps every DEK forward if the KEK version changed.

KeyringMaintenanceConfig

services.AddKeyringMaintenance(cfg =>
{
    cfg.LeaderKey = "hexasync:my-service:keyring-maintenance";
    cfg.MaxDekAge = TimeSpan.FromDays(30);      // rotate monthly instead of the 180-day default
    cfg.Interval  = TimeSpan.FromHours(12);     // still ≪ MaxDekAge
});

Shortening MaxDekAge is always safe; retired DEKs stay in the keyring for decrypt, so old blobs keep reading. What it costs is keyring growth and re-encryption pressure if you also run force-reconcile.

Want it env-driven? Read it yourself at the call site — the library binds no env var for it:

cfg.MaxDekAge = TimeSpan.Parse(builder.Configuration["DATA_PROTECTION_MAX_DEK_AGE"] ?? "180.00:00:00");

The 180-day timer is the enforced bound, not the only trigger. A per-process invocation counter can rotate earlier to stay inside the AES-GCM nonce budget — that one is env-tunable via DATA_PROTECTION_EXPECTED_MAX_PODS (and DATA_PROTECTION_NONCE_THRESHOLD_OVERRIDE). It is defense-in-depth; set EXPECTED_MAX_PODS ≥ your HPA maxReplicas and leave the rotation policy to MaxDekAge.

Rotating the DEK (this section, automatic) is not rotating the KEK (manual, in your vault, and data-loss-class if you get retention wrong — see the ops runbook).

Rotating the local master (the Local KEK)

The kek_id a locally-wrapped DEK carries is derived from the master key: local:<12 hex>, a domain-separated SHA-256 prefix. That is deliberate — it means the id changes exactly when the key changes, so the maintenance re-wrap notices a new master on its own. You cannot swap the master while the id stays put, which is what used to turn a master swap into silent, total, unrecoverable data loss.

To rotate it, keep the old master reachable for as long as any DEK is still wrapped by it:

# 1. Both masters present. The old one is a DECRYPT-ONLY fallback (token: LocalPrevious); it can never be KEK_ACTIVE.
DATA_PROTECTION_MASTER_KEY_BASE64=<the NEW master>
DATA_PROTECTION_MASTER_KEY_BASE64_OLD=<the master being retired>

2. Deploy. The next maintenance cycle re-wraps every DEK onto the new master (ReWrapAllAsync targets by exact kek_id), or call it directly for an immediate, logged pass. This requires AddKeyringMaintenance — without it nothing ever re-wraps and the rotation never completes.
3. Verify before step 4: AssertNoneWrappedByAsync("LocalPrevious") (fail-closed) or CountWrappedByAsync("LocalPrevious") == 0. The KEK status endpoint shows the same thing per kek_id.
4. Only now remove DATA_PROTECTION_MASTER_KEY_BASE64_OLD and deploy. Dropping it while step 3 is non-zero destroys those DEKs.

Already running Local today? Nothing to do and nothing to change. Existing rows carry the literal id from DATA_PROTECTION_KEK_ID (default local); they keep unwrapping — the AAD binds the id recorded on the row, not the current one — and they are re-wrapped onto the derived id on the first maintenance cycle. Keep DATA_PROTECTION_KEK_ID set to whatever it has been: it is how the drop gate still recognizes those rows as yours.

Composition refuses three configurations outright, each of which would break the rotation silently: KEK_ACTIVE pointing at LocalPrevious (new DEKs would be wrapped under the key you are retiring), ..._OLD identical to the current master (the drop gate could no longer tell them apart), and ..._OLD set with no current master.

DEK deletion & retention (how long a retired key is kept)

By default: forever. Rotation retires a DEK and keeps its material (KEK-wrapped) so old blobs still decrypt. Deletion is the separate opt-in AddDekDestroy path, which AddHexaSyncDataProtection does not wire. Not wiring it is a safe default — it only costs keyring rows.

Once wired, deletion is two-phase, on two independent clocks (DekDestroyConfig):

At defaults, measured from the moment the drain is confirmed: soft delete ≈ 30 days, hard delete ≈ 90 days. A leader evaluates both phases every CheckInterval (1 h + up to 5 min jitter), so each lands within about an hour of eligibility. Whole life of a DEK from creation ≈ 180 d rotation + drain + 30 d + 60 d ≈ 270+ days.

services.AddDekDestroy(cfg =>
{
    cfg.LeaderKey             = "hexasync:<your-service>:dek-destroy";  // per SERVICE
    cfg.DestroyGracePeriod    = TimeSpan.FromDays(30);   // → soft delete (recoverable)
    cfg.SoftDeleteRetention   = TimeSpan.FromDays(60);   // → hard shred (irreversible)
    cfg.CompromiseGracePeriod = TimeSpan.FromDays(1);    // → soft delete, compromised fast-path
});

Four behaviours worth knowing before you tune these:

• A compromised generation shortens the wait before the soft delete only. CompromiseGracePeriod (1 day, clamped so it can never lengthen the wait) replaces the 30 days, but the 60-day soft-delete window still applies — so material lifetime is ~61 days, not 1. Shorten SoftDeleteRetention too if you need it gone faster.
• The 30-day clock is retractable. retired_at is stamped only once the reconcile sweep confirms the generation is drained, and is reset to null if that confirmation drops — the 30 days restart. A generation with straggler rows never starts the clock. Consequence: without AddDekForceReconcile nothing is ever deleted (it is the only writer of the drain signal, which is why AddDekDestroy fails the host at startup without it, plus active-DEK fencing and a shared non-default IDekReconcileGate).
• Both phases re-run a fresh drained-veto at the instant they act. A row that reappears during the 60-day window (a backup restore, an unregistered dynamic table) defers the shred and logs it — it is never forced.
• RestoreAsync clears both timestamps, so a recovered generation must re-drain and serve the full DestroyGracePeriod again.

Lengthening either window is always safe. Shortening SoftDeleteRetention shortens your recovery runway — that window is what makes a premature soft-delete an operator fix instead of a data-loss event. Don't set it to zero.

Completing the lifecycle — the background services and what breaks without each

Seven hosted services ship here; AddHexaSyncDataProtection wires exactly one. The lifecycle is a chain: a missing stage doesn't error, it makes every later stage a silent no-op (destroy is the exception — it refuses to boot).

Most services should stop at stage 2. Stages 3-4 exist to retire key material; with no compliance driver for shredding, rotate + re-wrap and leave retired DEKs in the keyring — they cost rows, not risk.

Full wiring for a multi-pod service that deletes:

// Stage 1 (+ DATA_PROTECTION_ENABLE_ACTIVE_DEK_FENCING=true)
services.AddHexaSyncDataProtectionWithEf<MyDbContext>(configuration);
services.AddHealthChecks().AddHexaSyncDataProtectionHealthCheck();

// Your seams — multi-pod destroy needs the pointer cache AND the gate SHARED (Redis); the built-in defaults are
// per-pod and the boot guard rejects them.
services.AddSingleton<IDataProtectionLeaderLock, RedisMedallionLeaderLock>();
services.AddSingleton<IDekActiveKeyPointerCache, RedisDekActiveKeyPointerCache>();
services.AddSingleton<IDekReconcileGate, RedisDekReconcileGate>();
services.AddSingleton<IDekEvictionBroadcast, RedisDekEvictionBroadcast>();   // optional, recommended

// Stages 2-4 — each with its OWN per-service LeaderKey
services.AddKeyringMaintenance(cfg => { cfg.LeaderKey = "hexasync:my-service:keyring-maintenance"; });
services.AddDekForceReconcile<MyDbContext>(cfg =>
{
    cfg.LeaderKey = "hexasync:my-service:dek-force-reconcile";
    cfg.ActiveKeyPropagationDelay = TimeSpan.FromHours(24);   // REQUIRED for multi-pod destroy — see below
});
services.AddDekDestroy(cfg => { cfg.LeaderKey = "hexasync:my-service:dek-destroy"; });

Traps:

• ActiveKeyPropagationDelay defaults to TimeSpan.Zero (hold off) and nothing validates it. Multi-pod + AddDekDestroy must set it positive (24 h is the reference value), or rows can be reconciled onto a new key — arming destruction — while a stale pod still encrypts under the previous one. The boot guard checks fencing, the pointer cache and the gate; not this knob, so a misconfiguration boots clean.
• Every leader-elected Add… defaults LeaderKey to the same hexasync:dataprotection:* string — two services on one Redis then elect a leader between them and only one keyring gets maintained. Override all three.
• AddDekForceReconcile<TContext> needs AddDbContextFactory<TContext> — the drain is a singleton driver creating a fresh context per batch.

DekForceReconcileConfig

Lowering BatchSize or lengthening CheckInterval is always safe — the drain just takes longer. The one value that changes correctness rather than speed is ActiveKeyPropagationDelay.

• A drain pass that hits a fail-closed decrypt does not set the gate, so the next tick retries instead of declaring the generation drained — a stuck row blocks deletion rather than risking it.

Your DbContext

Implement IDataProtectionDbContext (exposes the keyring DbSet), inject the resolved IDataProtector into the ctor, and call ApplyDataProtection in OnModelCreating:

public class MyDbContext(DbContextOptions options, IDataProtector protector)
    : DbContext(options), IDataProtectionDbContext
{
    public DbSet<DataEncryptionKeyRecord> DataEncryptionKeys => Set<DataEncryptionKeyRecord>();
    public DbSet<UserSecret> UserSecrets => Set<UserSecret>();

    protected override void OnModelCreating(ModelBuilder mb) => mb.ApplyDataProtection(protector);
    // ApplyDataProtection wires the [Encrypted] converter, guards (no index/non-string [Encrypted]), and maps the
    // data_encryption_keys table by construction.
}

[Table("user_secrets")]
public class UserSecret
{
    [Key] public int Id { get; set; }
    [Encrypted] public string TotpSeed { get; set; } = "";   // stored as an hxdp: blob, transparently en/decrypted
    public string Label { get; set; } = "";                  // plaintext, queryable
}

Migration (you own it — the library ships none)

The keyring table your migration must create:

CREATE TABLE data_encryption_keys (
    key_id               text        PRIMARY KEY,
    wrapped_dek          bytea       NOT NULL,
    kek_id               text        NOT NULL,
    state                text        NOT NULL,   -- pending | active | retired | destroyed  (plain text, not an enum)
    created_under_scheme text        NOT NULL,   -- e.g. 'v1'
    created_at           timestamptz NOT NULL
);

Each [Encrypted] business column is text (it holds the ASCII hxdp: blob — not bytea). If an entity implements IEncryptedEntity (for the future 1.9 re-encrypt sweep), also add a nullable dek_reconciled_version text column. Do not put an index / WHERE / ORDER BY / FK on an [Encrypted] column (see the ops runbook).

Health check

AddHexaSyncDataProtectionHealthCheck() reports Unhealthy until the keyring warms (IsInitialized) — wire it to readiness so a pod that can't reach the KEK at boot doesn't take traffic. When AddKeyringMaintenance is wired, an overdue rotation (age > MaxDekAge + OverdueGrace) reports Degraded (never Unhealthy — the pod still serves). Without AddKeyringMaintenance there is no overdue check at all — a stuck rotation is invisible to health.

Migrating off a legacy format

The library is hxdp:-only; migrating legacy AES-CBC (or any prior format) is the consumer's job. Read the raw legacy value with your own code (raw SQL, or a temporary non-[Encrypted] mapping — you can't read it through the [Encrypted] property, which would try to decrypt it as hxdp: and fail closed), decrypt it with your legacy code, then write the plaintext through the [Encrypted] property so the converter re-encrypts it on save. Use HxdpEnvelope.IsProtected on the raw value to skip already-migrated rows:

foreach (var (id, raw) in ReadRawLegacyColumn())          // your code — bypasses the [Encrypted] converter
{
    if (HxdpEnvelope.IsProtected(raw)) continue;          // already "hxdp:" → migrated; leave it

    // Attach a stub and mark ONLY the [Encrypted] property modified. This emits an UPDATE that encrypts the
    // assigned plaintext — WITHOUT a prior SELECT, so the legacy value is never read through the converter
    // (which would fail closed). Do NOT db.Find/materialize the row here.
    var stub = new UserSecret { Id = id };
    db.Attach(stub);
    stub.TotpSeed = DecryptLegacy(raw);                   // plaintext in → converter encrypts on save
}
db.SaveChanges();

Non-EF callers use IDataProtector directly — note it is bytes-in / bytes-out (the result is the ASCII hxdp: blob): var blob = Encoding.ASCII.GetString(await protector.EncryptAsync(Encoding.UTF8.GetBytes(plaintext)));.

No-break rollout checklist

The failure mode of any migration is a pod (or a restored backup) that can't read what another pod already wrote. Order the rollout so that can't happen:

1. Additive schema first. Add the new [Encrypted]/v2 column; never repurpose the legacy column in the same deploy — legacy readers keep working.
2. Decrypt-before-encrypt. Deploy the version that can decrypt hxdp: to 100% of replicas before any version writes hxdp:. For the length of a rolling update the old and new versions are both Ready and serving; the old pod must not choke on a blob the new pod just wrote. (A config-flag cutover is the same — the flip lands pod-by-pod.)
3. Backfill idempotently. Skip already-migrated rows with HxdpEnvelope.IsProtected(raw); the job must be safe to re-run and to interrupt mid-flight.
4. Drop the legacy column only after prod backfill is confirmed — production-gated, a separate later deploy, never bundled with the write cutover.

For a KEK provider cutover (Local ↔ Azure), the same discipline is re-wrap-before-drop — see the next section.

Sensitive-data-logging prohibition

EnableSensitiveDataLogging would write decrypted plaintext + key material to logs. Call context.WarnIfSensitiveDataLoggingWithEncryptedProperties(logger) at startup — it logs a warning if the trap is on for a context with [Encrypted] properties. (It warns; it does not block.)

Observability

Metrics ship on a Meter named HexaSync.DataProtection (tag-free counters: encrypt/decrypt/decrypt_miss/ rotation/rewrap/kek_unwrap; gauges: active-DEK nonce consumption + age). Subscribe your OpenTelemetry exporter to that meter name. For per-event auditing (rotate / re-wrap / destroy / decrypt-miss / kek-version-change), implement IDataProtectionAuditSink and register it before AddHexaSyncDataProtection.

Crypto inventory

EncryptedFieldInventory.ToMarkdown(ctx.Model) renders the declared [Encrypted] fields as a Markdown table for a CRYPTO-INVENTORY.md artifact (emit it at boot or from a CLI dump mode). Schema metadata only — never key material.

Name note: IDataProtector

This library's hexasync.dataprotection.IDataProtector shares its simple name with Microsoft.AspNetCore.DataProtection.IDataProtector. If a consumer uses both, namespace-qualify or alias (using IHxDataProtector = hexasync.dataprotection.IDataProtector;). They are unrelated types.

Multiple KEK providers & seamless Local→Azure cutover

Both KEK providers can be registered at once (a Local master and AddAzureKeyVaultKeyWrapper). They compose behind a single IKeyWrapper:

• Wrap (new + re-wrapped DEKs) always uses the active provider (DATA_PROTECTION_KEK_ACTIVE).
• Unwrap tries the active provider first; on a wrong-key/wrong-format failure it falls back to the other provider (so a DEK wrapped by either is always decryptable through a migration). A transient failure (KV throttle/timeout/5xx) retries the active provider and never falls through — an outage is never masked. A KEK outage degrades rotation, not decryption of already-cached DEKs.

Cutover (Local → Azure), reversible until the last step:

// 0. Register BOTH: Local master present + AddAzureKeyVaultKeyWrapper(...); DATA_PROTECTION_KEK_ACTIVE=Local.
// 1. Prove Azure works while Local is still active — the self-test endpoint (below), green across the fleet.
// 2. Flip DATA_PROTECTION_KEK_ACTIVE=AzureKeyVault  → new wraps use Azure; existing DEKs still unwrap via Local.
// 3. Re-wrap the existing DEK(s) onto the active provider (Local → Azure), no data re-encryption:
await provider.ReWrapAllToActiveAsync(ct);        // provider = the resolved KeyringDekProvider
// 4. Soak. Rollback (before step 5) = flip active back + ReWrapAllToActiveAsync again.
// 5. Drop Local ONLY after the fail-closed gate passes (else a still-Local-wrapped DEK becomes unrecoverable):
await provider.AssertNoneWrappedByAsync("Local", ct);   // throws if any DEK is still Local-wrapped
//    → then remove DATA_PROTECTION_MASTER_KEY_BASE64.

Both providers stay registered throughout, so a pod restart mid-cutover always unwraps. The gate classifies each DEK's actual recorded kek_id (not the literal "local"), so a custom DATA_PROTECTION_KEK_ID still gates correctly.

KEK self-test endpoint (hexasync.dataprotection.endpoints)

An admin can confirm a named provider's full path (auth/network/RBAC/wrap/unwrap) on demand — e.g. prove Azure works while Local is still active — without touching the keyring or real data:

builder.Services.AddDataProtectionSelfTestRateLimiter();           // fixed-window, 5/min per caller
app.UseRateLimiter();
// Authorization is CONSUMER-supplied via the configure callback — this package depends on no auth middleware
// (it is upstream of every consumer), so you apply your own capability gate here:
app.MapDataProtectionSelfTest("/api/v3/admin/dataprotection/self-test",
    endpoint => endpoint.RequireCapability(DataProtectionCapabilities.KekSelfTest));
// POST .../self-test?provider=AzureKeyVault  →  { provider, ok, kekIdFingerprint, latencyMs, errorKind? }

DataProtectionCapabilities.KekSelfTest ("dataprotection:kek:self_test") is a library-owned string constant — mirror it into your capability catalog and grant it to your admin/super-admin role. The response returns a fingerprint of the kek_id (never the vault URI) and a classified errorKind (never raw provider exception text). The self-test calls the adapter directly — it writes nothing to the keyring.

Operating the key lifecycle — every lever, and how to smoke-test it

Everything below is opt-in. Wire the endpoints you want; a surface you do not map does not exist. Every endpoint takes a required configure callback where you apply your own capability gate — this package depends on no auth middleware.

The full surface

Automatic, once AddKeyringMaintenance is wired: bootstrap the first DEK, rotate at MaxDekAge, and re-wrap onto the current KEK — all leader-elected. The manual levers exist for the cases a timer cannot serve: a suspected exposure, a rehearsal, a cutover you want logged and watched.

Smoke-testing the whole lifecycle locally

Run this against a local instance with Local custody — no vault needed. It exercises both lifecycles end to end; each step has an observable result, so a failure tells you which mechanism broke.

BASE=http://localhost:5000/api/v3/admin/dataprotection
AUTH="Authorization: Bearer $TOKEN"          # a principal holding the four capabilities above

# 1. KEK reachable? (no keyring writes, no real data)
curl -sX POST "$BASE/self-test?provider=Local" -H "$AUTH"     # → { ok: true, kekIdFingerprint, latencyMs }

# 2. Keyring bootstrapped at boot: exactly one active DEK, everything on the current KEK
curl -s "$BASE/kek-status" -H "$AUTH"                          # → stale: 0, byKek: [ { isCurrent: true, … } ]

# 3. Encrypt something through the app, then read the column: it must start `hxdp:v1.<key_id>.`
#    — that key_id is the generation, in the clear, by design.

# 4. Rotate the DEK. New generation; the previous is RETIRED and retained, so step-3 data still decrypts.
curl -sX POST "$BASE/dek/rotate" -H "$AUTH"                    # → { keyId: "<new>" }
#    Re-read the step-3 row (still decrypts, still its old key_id) and write a new row (new key_id).

# 5. Which tables still sit on the old generation?
curl -s "$BASE/dek-status" -H "$AUTH"

Then the KEK half — swap the local master, which is the part with a real failure mode:

# 6. Set DATA_PROTECTION_MASTER_KEY_BASE64_OLD=<current>, DATA_PROTECTION_MASTER_KEY_BASE64=<new>. Restart.
curl -s "$BASE/kek-status" -H "$AUTH"     # → every DEK still readable, now reported against TWO kek_ids

# 7. Let maintenance run (or call ReWrapAllAsync). Re-check: one kek_id, stale 0.
# 8. Prove the old master is unused BEFORE deleting it:
#    AssertNoneWrappedByAsync("LocalPrevious") — throws if any DEK still needs it.
# 9. Only now remove DATA_PROTECTION_MASTER_KEY_BASE64_OLD and restart.

Step 8 is the one step you cannot skip: dropping a master while a live DEK is still wrapped by it is unrecoverable. The gate is fail-closed for that reason.

What this does not cover. Destruction is deliberately slow and cannot be smoke-tested in one sitting — a retired DEK waits out a quarantine window, then a recoverable soft-delete window, with a fresh drain-veto re-checked at the shred instant. Rehearse it with shortened windows in a scratch environment, never by shortening them in production.

Keep this as a checklist you run by hand against a local instance. It is not a CI job: it asserts operator-visible behaviour of a running deployment, and a green run in a container proves nothing about a cluster.

Operations

KEK-retention, compromise/rotation, encrypted-column rules, and the KV rotation burst budget are in the ops runbook: docs/data-protection/OPS_RUNBOOK.md. The KEK-retention rule is data-loss-class — read it before rotating a KEK in production.