hexasync.dataprotection

Envelope encryption (AES-256-GCM) with pluggable KEK custody and key rotation — the HexaSync at-rest data-protection library. Self-describing hxdp: wire format, code-registry algorithm versioning, per-service DEK keyring with in-memory cache, EF Core integration ([Encrypted]), and a self-healing DEK/KEK lifecycle driver.

The Azure Key Vault KEK adapter ships separately as hexasync.dataprotection.azurekeyvault.

How it works (30 seconds)

A KEK (key-encryption key) wraps per-generation DEKs (data-encryption keys). With AzureKeyVault there is one KEK per environment, shared by every service in it; with Local the KEK is the service's own master key (DATA_PROTECTION_MASTER_KEY_BASE64). Your data is encrypted with the active DEK; the wrapped DEKs live in a data_encryption_keys table in your database. On boot (EnsureDataProtectionReadyAsync, see Wiring) the library proves the active KEK with a wrap→unwrap round-trip, then unwraps every DEK row that is neither destroyed nor soft-deleted (one KEK unwrap per row) into an in-memory cache, minting the first DEK on an empty keyring. If that still fails after DATA_PROTECTION_STARTUP_MAX_ATTEMPTS, the process does not start. From then on encrypt/decrypt are local, and a per-pod keyring follower picks up DEK generations other pods mint. It is hxdp:-only and fails closed — it never falls back to plaintext or a legacy format.

Install

dotnet add package hexasync.dataprotection
dotnet add package hexasync.dataprotection.azurekeyvault   # production KEK custody (optional in dev)

Configuration (flat DATA_PROTECTION_* env vars)

House convention: config via flat UPPER_SNAKE_CASE environment variables (no appsettings.json). Each maps to a property via [ConfigurationKeyName], bound as IOptions<DataProtectionConfig>.

DATA_PROTECTION_KEK_PROVIDER=Local              # "Local" (dev/low envs) or "AzureKeyVault" (prod); also the default KEK_ACTIVE
DATA_PROTECTION_KEK_ACTIVE=Local                # which registered provider WRAPS new/re-wrapped DEKs (default = KEK_PROVIDER).
                                                 # Register BOTH providers and flip this for a seamless Local↔Azure cutover
                                                 # (see "Multiple KEK providers" below). Unwrap always routes to whoever wrapped a DEK.
DATA_PROTECTION_MASTER_KEY_BASE64=<b64 32 bytes> # REQUIRED when Local is active/fallback (`openssl rand -base64 32`)
DATA_PROTECTION_MASTER_KEY_BASE64_OLD=          # the PREVIOUS master, set only while rotating it (see "Rotating the local master")
DATA_PROTECTION_KEK_ID=local                    # the id LEGACY rows carry. It no longer stamps new wraps — the kek_id is
                                                 # derived from the master itself (`local:<12 hex>`), so it tracks the key
DATA_PROTECTION_ACTIVE_SCHEME=v1                # optional; default = newest registered scheme; unknown → boot throws
DATA_PROTECTION_EXPECTED_MAX_PODS=16            # must be >= your HPA maxReplicas; drives the nonce early-rotation trip
DATA_PROTECTION_STARTUP_MAX_ATTEMPTS=5          # optional; bounded boot-gate attempts (KEK / database may be transiently
                                                 # down). Exhaustion STOPS THE PROCESS — the gate throws, the host does not start
DATA_PROTECTION_STARTUP_RETRY_DELAY=00:00:02    # optional; base backoff between attempts, doubling each time (capped at 30 s)
DATA_PROTECTION_FOLLOW_INTERVAL=00:01:00        # optional; how often every pod's keyring follower adopts DEK generations
                                                 # minted elsewhere (fencing off; with fencing on it refreshes at pointer TTL/2)

Local is for dev / low environments. The master key sits in config; a KV-wrapped DEK cannot be unwrapped locally (by design — prod data is not decryptable off-cluster). Production should use AzureKeyVault (see that package's README).

Key rotation has no env var. The DEK rotation period and the rest of the maintenance schedule are configured in code via the AddKeyringMaintenance(cfg => …) lambda — there is no DATA_PROTECTION_MAX_DEK_AGE and no DataProtection:KeyringMaintenance JSON section. See Key rotation.

Wiring (EF-backed, in order)

// 1. Your DbContext factory — the library needs a FRESH context per keyring call (the provider is a singleton;
//    a captured scoped DbContext would throw ObjectDisposedException under load).
services.AddDbContextFactory<MyDbContext>(o => o.UseNpgsql(connectionString));

// 2. Compose the library (store + provider + IDataProtector + boot gate + keyring follower); binds the
//    DATA_PROTECTION_* env vars. It also registers a hosted initializer — a FALLBACK that runs the boot gate if
//    step 6 is skipped, but only in hosted-service registration order, so nothing guarantees it runs first.
services.AddHexaSyncDataProtectionWithEf<MyDbContext>(configuration);   // bundles AddEfDekKeyringStore<MyDbContext>()

// 3. Azure Key Vault KEK custody — only where a vault is configured: AddAzureKeyVaultKeyWrapper THROWS when
//    DATA_PROTECTION_KV_VAULT_URI or DATA_PROTECTION_KV_KEY_NAME is unset, so an unconditional call breaks every
//    environment that runs Local only. May be registered ALONGSIDE Local (both coexist — DATA_PROTECTION_KEK_ACTIVE
//    picks which one wraps; unwrap routes to whichever wrapped a given DEK). This is the seamless-cutover path (below).
if (!string.IsNullOrWhiteSpace(configuration["DATA_PROTECTION_KV_VAULT_URI"]))
    services.AddAzureKeyVaultKeyWrapper(configuration);

// 4. Health check (readiness): after AddHexaSyncDataProtection. Optional failureStatus — see "Health check".
services.AddHealthChecks().AddHexaSyncDataProtectionHealthCheck(tags: ["ready"]);

// 5. Opt-in self-healing maintenance (DEK rotation + KEK re-wrap). Requires a leader lock (below).
//    WITHOUT this call nothing ever rotates or re-wraps — see "Key rotation" for what that costs you.
services.AddSingleton<IDataProtectionLeaderLock, RedisMedallionLeaderLock>();
services.AddKeyringMaintenance(cfg =>
{
    cfg.LeaderKey = "hexasync:<your-service>:keyring-maintenance";   // per SERVICE — the default is shared
    cfg.MaxDekAge = TimeSpan.FromDays(180);                          // the DEK rotation period (this is the default)
});

// 6. Program.cs, after builder.Build(): migrate → boot gate → run. The gate needs the keyring table, so it goes
//    AFTER your migrations, and BEFORE app.Run() so nothing touching encrypted data runs ahead of it.
var app = builder.Build();
await using (var db = await app.Services.GetRequiredService<IDbContextFactory<MyDbContext>>().CreateDbContextAsync())
    await db.Database.MigrateAsync();
await app.Services.EnsureDataProtectionReadyAsync();   // throws after STARTUP_MAX_ATTEMPTS → the process does not start
app.Run();

EnsureDataProtectionReadyAsync() blocks until this pod can encrypt: the active KEK passes a wrap→unwrap round-trip, the keyring is loaded (first DEK minted on an empty keyring) and, with fencing on, the active-DEK lease is installed. It retries with backoff, then throws DataProtectionException. A KEK or database that is unreachable at boot therefore stops the process — restarting or scaling pods do not come up until it returns, while pods already running keep serving from their cache. Later calls return without I/O. The reasoning behind the explicit call is in the consumer guide, docs/data-protection/README.md in the repository.

A custom audit sink (optional) must be registered before AddHexaSyncDataProtection (it wins a TryAdd over the default no-op): services.AddSingleton<IDataProtectionAuditSink, MySink>();.

Leader-lock adapter (Medallion.Threading Redis)

AddKeyringMaintenance (and AddDekForceReconcile, AddDekDestroy) needs an IDataProtectionLeaderLock so one replica per service leads each loop. The library ships no Redis dependency — adapt your existing distributed lock (the shape is a single blocking acquire returning a handle held until disposed). The handle may also implement IDataProtectionLeaderLease to report a lost lease; AcquireAsync's signature is the same either way:

public sealed class RedisMedallionLeaderLock(IDistributedLockProvider locks) : IDataProtectionLeaderLock
{
    public async Task<IAsyncDisposable> AcquireAsync(string key, CancellationToken ct)
        => new Lease(await locks.AcquireLockAsync(key, cancellationToken: ct));

    // The handle the leader loops hold and dispose. Optional: returning Medallion's handle directly also works,
    // but then a loop never learns that its lease was lost.
    private sealed class Lease(IDistributedSynchronizationHandle inner) : IDataProtectionLeaderLease
    {
        public CancellationToken LeaseLost => inner.HandleLostToken;   // read lazily — accessing it may start monitoring
        public ValueTask DisposeAsync() => inner.DisposeAsync();
    }
}

AcquireAsync should block until acquired (so a dead leader's lock is picked up as soon as it releases).

Exclusivity is best-effort, not guaranteed. A lease can be lost while its holder still runs (a Redis failover or a partition that outlasts it), and a second replica then acquires. With a handle that implements IDataProtectionLeaderLease (and a LeaseLost that can be cancelled), each leader loop steps down once the provider reports the loss, waits its ReacquireDelay and acquires again — but until the report arrives both replicas lead (Medallion's Redis lock notices at its next lease extension). With a plain handle a stale leader keeps leading until it stops or its loop errors. Correctness never rests on exclusivity: maintenance and reconcile are idempotent, and destroy does not depend on the lease — its safety is its guarded compare-and-swap writes, the fresh drain veto at the instant it acts, the soft-delete window and active-DEK fencing, so a plain handle is safe for destroy too. Detection delay and tuning: the consumer guide, docs/data-protection/README.md in the repository.

Key rotation (DEK age) and the maintenance cycle

The first DEK is created at startup regardless — but nothing rotates until you call AddKeyringMaintenance. AddHexaSyncDataProtection deliberately does not wire it (that would force an IDataProtectionLeaderLock on every consumer). Skip it and you get a DEK that encrypts forever, no KEK re-wrap when the KEK version changes, and no overdue signal — the health check's overdue branch only runs when KeyringMaintenanceConfig is registered (its other Degraded condition, a stale keyring follower, does not need it).

One elected leader per service runs a cycle every Interval, and each cycle: ensures an active DEK exists → rotates it if it is older than MaxDekAge (the previous DEK is retained so old data still decrypts) → re-wraps every DEK forward if the KEK version changed.

KeyringMaintenanceConfig

services.AddKeyringMaintenance(cfg =>
{
    cfg.LeaderKey = "hexasync:my-service:keyring-maintenance";
    cfg.MaxDekAge = TimeSpan.FromDays(30);      // rotate monthly instead of the 180-day default
    cfg.Interval  = TimeSpan.FromHours(12);     // still ≪ MaxDekAge
});

Shortening MaxDekAge is always safe; retired DEKs stay in the keyring for decrypt, so old blobs keep reading. What it costs is keyring growth and re-encryption pressure if you also run force-reconcile.

Want it env-driven? Read it yourself at the call site — the library binds no env var for it:

cfg.MaxDekAge = TimeSpan.Parse(builder.Configuration["DATA_PROTECTION_MAX_DEK_AGE"] ?? "180.00:00:00");

The 180-day timer is the enforced bound, not the only trigger. A per-process invocation counter can rotate earlier to stay inside the AES-GCM nonce budget — that one is env-tunable via DATA_PROTECTION_EXPECTED_MAX_PODS (and DATA_PROTECTION_NONCE_THRESHOLD_OVERRIDE). It is defense-in-depth; set EXPECTED_MAX_PODS ≥ your HPA maxReplicas and leave the rotation policy to MaxDekAge.

Rotating the DEK (this section, automatic) is not rotating the KEK (manual, in your vault, and data-loss-class if you get retention wrong — see the ops runbook).

Rotating the local master (the Local KEK)

The kek_id a locally-wrapped DEK carries is derived from the master key: local:<12 hex>, a domain-separated SHA-256 prefix. That is deliberate — it means the id changes exactly when the key changes, so the maintenance re-wrap notices a new master on its own. You cannot swap the master while the id stays put, which is what used to turn a master swap into silent, total, unrecoverable data loss.

To rotate it, keep the old master reachable for as long as any DEK is still wrapped by it:

# 1. Both masters present. The old one is a DECRYPT-ONLY fallback (token: LocalPrevious); it can never be KEK_ACTIVE.
DATA_PROTECTION_MASTER_KEY_BASE64=<the NEW master>
DATA_PROTECTION_MASTER_KEY_BASE64_OLD=<the master being retired>

2. Deploy. The next maintenance cycle re-wraps every DEK onto the new master (ReWrapAllAsync targets by exact kek_id), or call it directly for an immediate, logged pass. This requires AddKeyringMaintenance — without it nothing ever re-wraps and the rotation never completes.
3. Verify before step 4: AssertNoneWrappedByAsync("LocalPrevious") (fail-closed) or CountWrappedByAsync("LocalPrevious") == 0. The KEK status endpoint shows the same thing per kek_id.
4. Only now remove DATA_PROTECTION_MASTER_KEY_BASE64_OLD and deploy. Dropping it while step 3 is non-zero destroys those DEKs.

Already running Local today? Nothing to do and nothing to change. Existing rows carry the literal id from DATA_PROTECTION_KEK_ID (default local); they keep unwrapping — the AAD binds the id recorded on the row, not the current one — and they are re-wrapped onto the derived id on the first maintenance cycle. Keep DATA_PROTECTION_KEK_ID set to whatever it has been: it is how the drop gate still recognizes those rows as yours.

Composition refuses three configurations outright, each of which would break the rotation silently: KEK_ACTIVE pointing at LocalPrevious (new DEKs would be wrapped under the key you are retiring), ..._OLD identical to the current master (the drop gate could no longer tell them apart), and ..._OLD set with no current master.

DEK deletion & retention (how long a retired key is kept)

By default: forever. Rotation retires a DEK and keeps its material (KEK-wrapped) so old blobs still decrypt. Deletion is the separate opt-in AddDekDestroy path, which AddHexaSyncDataProtection does not wire. Not wiring it is a safe default — it only costs keyring rows.

Once wired, deletion is two-phase, on two independent clocks (DekDestroyConfig):

At defaults, measured from the moment the drain is confirmed: soft delete ≈ 30 days, hard delete ≈ 90 days. A leader evaluates both phases every CheckInterval (1 h + up to 5 min jitter), so each lands within about an hour of eligibility. Whole life of a DEK from creation ≈ 180 d rotation + drain + 30 d + 60 d ≈ 270+ days.

services.AddDekDestroy(cfg =>
{
    cfg.LeaderKey             = "hexasync:<your-service>:dek-destroy";  // per SERVICE
    cfg.DestroyGracePeriod    = TimeSpan.FromDays(30);   // → soft delete (recoverable)
    cfg.SoftDeleteRetention   = TimeSpan.FromDays(60);   // → hard shred (irreversible)
    cfg.CompromiseGracePeriod = TimeSpan.FromDays(1);    // → soft delete, compromised fast-path
});

Four behaviours worth knowing before you tune these:

• A compromised generation shortens the wait before the soft delete only. CompromiseGracePeriod (1 day, clamped so it can never lengthen the wait) replaces the 30 days, but the 60-day soft-delete window still applies — so material lifetime is ~61 days, not 1. Shorten SoftDeleteRetention too if you need it gone faster.
• The 30-day clock is retractable. retired_at is stamped only once the reconcile sweep confirms the generation is drained, and is reset to null if that confirmation drops — the 30 days restart. A generation with straggler rows never starts the clock. Consequence: without AddDekForceReconcile nothing is ever deleted — it is the only writer of the drain signal. AddDekDestroy's startup guard checks active-DEK fencing, a shared pointer cache and a shared non-default IDekReconcileGate; it does not check for AddDekForceReconcile, so destroy wired without it boots and silently deletes nothing.
• Both phases re-run a fresh drained-veto at the instant they act. A row that reappears during the 60-day window (a backup restore, an unregistered dynamic table) defers the shred and logs it — it is never forced.
• RestoreAsync clears both timestamps, so a recovered generation must re-drain and serve the full DestroyGracePeriod again.

Lengthening either window is always safe. Shortening SoftDeleteRetention shortens your recovery runway — that window is what makes a premature soft-delete an operator fix instead of a data-loss event. Don't set it to zero.

Completing the lifecycle — the background services and what breaks without each

Seven hosted services ship here; AddHexaSyncDataProtection wires two — the boot-gate initializer and the keyring follower. The lifecycle is a chain: a missing stage doesn't error, it makes every later stage a silent no-op (destroy is the exception — it refuses to boot).

Most services should stop at stage 2. Stages 3-4 exist to retire key material; with no compliance driver for shredding, rotate + re-wrap and leave retired DEKs in the keyring — they cost rows, not risk.

Full wiring for a multi-pod service that deletes:

// Stage 1 (+ DATA_PROTECTION_ENABLE_ACTIVE_DEK_FENCING=true)
services.AddHexaSyncDataProtectionWithEf<MyDbContext>(configuration);
services.AddHealthChecks().AddHexaSyncDataProtectionHealthCheck();
// …and in Program.cs: migrate, then await app.Services.EnsureDataProtectionReadyAsync(), then app.Run() — see Wiring.

// Your seams — multi-pod destroy needs the pointer cache AND the gate SHARED (Redis); the built-in defaults are
// per-pod and the boot guard rejects them.
services.AddSingleton<IDataProtectionLeaderLock, RedisMedallionLeaderLock>();
services.AddSingleton<IDekActiveKeyPointerCache, RedisDekActiveKeyPointerCache>();
services.AddSingleton<IDekReconcileGate, RedisDekReconcileGate>();
services.AddSingleton<IDekEvictionBroadcast, RedisDekEvictionBroadcast>();   // optional, recommended

// Stages 2-4 — each with its OWN per-service LeaderKey
services.AddKeyringMaintenance(cfg => { cfg.LeaderKey = "hexasync:my-service:keyring-maintenance"; });
services.AddDekForceReconcile<MyDbContext>(cfg =>
{
    cfg.LeaderKey = "hexasync:my-service:dek-force-reconcile";
    cfg.ActiveKeyPropagationDelay = TimeSpan.FromHours(24);   // REQUIRED for multi-pod destroy — see below
});
services.AddDekDestroy(cfg => { cfg.LeaderKey = "hexasync:my-service:dek-destroy"; });

Traps:

• ActiveKeyPropagationDelay defaults to TimeSpan.Zero (hold off) and nothing validates it. Multi-pod + AddDekDestroy must set it positive (24 h is the reference value), or rows can be reconciled onto a new key — arming destruction — while a stale pod still encrypts under the previous one. The boot guard checks fencing, the pointer cache and the gate; not this knob, so a misconfiguration boots clean.
• Every leader-elected Add… defaults LeaderKey to the same hexasync:dataprotection:* string — two services on one Redis then elect a leader between them and only one keyring gets maintained. Override all three.
• AddDekForceReconcile<TContext> needs AddDbContextFactory<TContext> — the drain is a singleton driver creating a fresh context per batch.

DekForceReconcileConfig

Lowering BatchSize or lengthening CheckInterval is always safe — the drain just takes longer. The one value that changes correctness rather than speed is ActiveKeyPropagationDelay.

• A drain pass that hits a fail-closed decrypt does not set the gate, so the next tick retries instead of declaring the generation drained — a stuck row blocks deletion rather than risking it.

Your DbContext

Implement IDataProtectionDbContext (exposes the keyring DbSet), inject the resolved IDataProtector into the ctor, and call ApplyDataProtection in OnModelCreating:

public class MyDbContext(DbContextOptions options, IDataProtector protector)
    : DbContext(options), IDataProtectionDbContext
{
    public DbSet<DataEncryptionKeyRecord> DataEncryptionKeys => Set<DataEncryptionKeyRecord>();
    public DbSet<UserSecret> UserSecrets => Set<UserSecret>();

    protected override void OnModelCreating(ModelBuilder mb) => mb.ApplyDataProtection(protector);
    // ApplyDataProtection wires the [Encrypted] converter, guards (no index/non-string [Encrypted]), and maps the
    // data_encryption_keys table by construction.
}

[Table("user_secrets")]
public class UserSecret : IEncryptedEntity
{
    [Key] public int Id { get; set; }
    [Encrypted] public string TotpSeed { get; set; } = "";   // stored as an hxdp: blob, transparently en/decrypted
    public string Label { get; set; } = "";                  // plaintext, queryable
    public string? DekReconciledVersion { get; set; }        // the watermark — written by the library only, never by you
}

Every entity with an [Encrypted] column implements IEncryptedEntity: the force-reconcile drain and the destroy gate see only tables that carry the DekReconciledVersion watermark, so a row without it is never moved to the current DEK and cannot stop the destruction of the DEK it is still encrypted under.

Migration (you own it — the library ships none)

The keyring table's schema comes from the model, not from this README. ApplyDataProtection calls ModelBuilderExtensions.ConfigureDataEncryptionKeys(), which maps the data_encryption_keys table and its columns — but only if your context already maps DataEncryptionKeyRecord. The DataEncryptionKeys DbSet that IDataProtectionDbContext requires is what puts it in the model; without it, ApplyDataProtection skips the keyring and the scaffold creates no table. A keyring-only context (no [Encrypted] columns) calls mb.ConfigureDataEncryptionKeys() itself.

Scaffold the migration with dotnet ef migrations add <Name>; never hand-write the keyring DDL. EF/ModelBuilderExtensions.cs is the source of truth, and a hand-written copy drifts from it as columns are added.

Each [Encrypted] business column is text (it holds the ASCII hxdp: blob — not bytea). For an entity that implements IEncryptedEntity, the model maps the DekReconciledVersion watermark column and ApplyDataProtection adds its index, so the scaffold generates both — do not add them by hand. Do not put an index / WHERE / ORDER BY / FK on an [Encrypted] column (see the ops runbook).

Health check

AddHexaSyncDataProtectionHealthCheck(name = "data-protection", failureStatus = null, tags = null) reports whether this pod can encrypt — wire it to readiness. It reads cache-only state (no store or KEK round-trip) and never reports key material. In order, first match wins:

1. Keyring not initialized → the failure status.
2. No active DEK in the cache (fencing off), or an expired active-DEK lease (fencing on) — encryption fails → the failure status.
3. Fencing off: the keyring follower has not succeeded for more than 3 × FollowInterval (new DEK generations are not being picked up) → Degraded.
4. Active-DEK rotation overdue (age > MaxDekAge + OverdueGrace) → Degraded, never the failure status — the pod still serves. Only when AddKeyringMaintenance is wired: without it there is no overdue check at all — a stuck rotation is invisible to health.
5. Otherwise Healthy.

The failure status is the registration's failureStatus, Unhealthy by default. A pod that cannot encrypt still decrypts every generation in its cache; to keep that decrypt-only traffic in the load balancer, register with Degraded:

services.AddHealthChecks().AddHexaSyncDataProtectionHealthCheck(
    failureStatus: HealthStatus.Degraded,   // null (default) → Unhealthy
    tags: ["ready"]);                        // passed to AddCheck, e.g. to filter a readiness endpoint

A boot that cannot reach the KEK or the keyring stops the process (the boot gate, see Wiring), so this check covers what happens after boot.

Migrating off a legacy format

The library is hxdp:-only; migrating legacy AES-CBC (or any prior format) is the consumer's job. Read the raw legacy value with your own code (raw SQL, or a temporary non-[Encrypted] mapping — you can't read it through the [Encrypted] property, which would try to decrypt it as hxdp: and fail closed), decrypt it with your legacy code, then write the plaintext through the [Encrypted] property so the converter re-encrypts it on save. Use HxdpEnvelope.IsProtected on the raw value to skip already-migrated rows:

foreach (var (id, raw) in ReadRawLegacyColumn())          // your code — bypasses the [Encrypted] converter
{
    if (HxdpEnvelope.IsProtected(raw)) continue;          // already "hxdp:" → migrated; leave it

    // Attach a stub and mark ONLY the [Encrypted] property modified. This emits an UPDATE that encrypts the
    // assigned plaintext — WITHOUT a prior SELECT, so the legacy value is never read through the converter
    // (which would fail closed). Do NOT db.Find/materialize the row here.
    var stub = new UserSecret { Id = id };
    db.Attach(stub);
    stub.TotpSeed = DecryptLegacy(raw);                   // plaintext in → converter encrypts on save
}
db.SaveChanges();

Non-EF callers use IDataProtector directly — note it is bytes-in / bytes-out (the result is the ASCII hxdp: blob): var blob = Encoding.ASCII.GetString(await protector.EncryptAsync(Encoding.UTF8.GetBytes(plaintext)));.

No-break rollout checklist

The failure mode of any migration is a pod (or a restored backup) that can't read what another pod already wrote. Order the rollout so that can't happen:

1. Additive schema first. Add the new [Encrypted]/v2 column; never repurpose the legacy column in the same deploy — legacy readers keep working.
2. Decrypt-before-encrypt. Deploy the version that can decrypt hxdp: to 100% of replicas before any version writes hxdp:. For the length of a rolling update the old and new versions are both Ready and serving; the old pod must not choke on a blob the new pod just wrote. (A config-flag cutover is the same — the flip lands pod-by-pod.)
3. Backfill idempotently. Skip already-migrated rows with HxdpEnvelope.IsProtected(raw); the job must be safe to re-run and to interrupt mid-flight.
4. Drop the legacy column only after prod backfill is confirmed — production-gated, a separate later deploy, never bundled with the write cutover.

For a KEK provider cutover (Local ↔ Azure), the same discipline is re-wrap-before-drop — see the next section.

Sensitive-data-logging prohibition

EnableSensitiveDataLogging would write decrypted plaintext + key material to logs. Call context.WarnIfSensitiveDataLoggingWithEncryptedProperties(logger) at startup — it logs a warning if the trap is on for a context with [Encrypted] properties. (It warns; it does not block.)

Observability

Metrics ship on a Meter named HexaSync.DataProtection (tag-free counters: encrypt/decrypt/decrypt_miss/ rotation/rewrap/kek_unwrap; gauges: active-DEK nonce consumption + age). Subscribe your OpenTelemetry exporter to that meter name. For per-event auditing (rotate / re-wrap / destroy / decrypt-miss / kek-version-change / self-test), implement IDataProtectionAuditSink and register it before AddHexaSyncDataProtection.

The boot gate emits self-test events. Each boot-gate attempt runs the active KEK's wrap→unwrap probe, and each probe that returns a result — pass or fail — emits one OnSelfTestAsync event, the same event the self-test endpoint emits. A boot that needs N attempts emits up to N (a probe that throws emits none). Expect them on every pod start.

Crypto inventory

EncryptedFieldInventory.ToMarkdown(ctx.Model) renders the declared [Encrypted] fields as a Markdown table for a CRYPTO-INVENTORY.md artifact (emit it at boot or from a CLI dump mode). Schema metadata only — never key material.

Name note: IDataProtector

This library's hexasync.dataprotection.IDataProtector shares its simple name with Microsoft.AspNetCore.DataProtection.IDataProtector. If a consumer uses both, namespace-qualify or alias (using IHxDataProtector = hexasync.dataprotection.IDataProtector;). They are unrelated types.

Multiple KEK providers & seamless Local→Azure cutover

Both KEK providers can be registered at once (a Local master and AddAzureKeyVaultKeyWrapper). They compose behind a single IKeyWrapper:

• Wrap (new + re-wrapped DEKs) always uses the active provider (DATA_PROTECTION_KEK_ACTIVE).
• Unwrap tries the active provider first; on a wrong-key/wrong-format failure it falls back to the other provider (so a DEK wrapped by either is always decryptable through a migration). A transient failure (KV throttle/timeout/5xx) retries the active provider and never falls through — an outage is never masked. A KEK outage degrades rotation, not decryption of already-cached DEKs.

Cutover (Local → Azure), reversible until the last step:

// 0. Register BOTH: Local master present + AddAzureKeyVaultKeyWrapper(...); DATA_PROTECTION_KEK_ACTIVE=Local.
// 1. Prove Azure works while Local is still active — the self-test endpoint (below), green across the fleet.
// 2. Flip DATA_PROTECTION_KEK_ACTIVE=AzureKeyVault  → new wraps use Azure; existing DEKs still unwrap via Local.
// 3. Re-wrap the existing DEK(s) onto the active provider (Local → Azure), no data re-encryption:
await provider.ReWrapAllToActiveAsync(ct);        // provider = the resolved KeyringDekProvider
// 4. Soak. Rollback (before step 5) = flip active back + ReWrapAllToActiveAsync again.
// 5. Drop Local ONLY after the fail-closed gate passes (else a still-Local-wrapped DEK becomes unrecoverable):
await provider.AssertNoneWrappedByAsync("Local", ct);   // throws if any DEK is still Local-wrapped
//    → then remove DATA_PROTECTION_MASTER_KEY_BASE64.

Both providers stay registered throughout, so a pod restart mid-cutover always unwraps. The gate classifies each DEK's actual recorded kek_id (not the literal "local"), so a custom DATA_PROTECTION_KEK_ID still gates correctly.

KEK self-test endpoint (hexasync.dataprotection.endpoints)

An admin can confirm a named provider's full path (auth/network/RBAC/wrap/unwrap) on demand — e.g. prove Azure works while Local is still active — without touching the keyring or real data:

builder.Services.AddDataProtectionSelfTestRateLimiter();           // fixed-window, 5/min per caller
app.UseAuthentication();
app.UseAuthorization();
app.UseRateLimiter();                                              // AFTER authentication — see below
// Authorization is CONSUMER-supplied via the configure callback — this package depends on no auth middleware
// (it is upstream of every consumer), so you apply your own capability gate here:
app.MapDataProtectionSelfTest("/api/v3/admin/dataprotection/self-test",
    endpoint => endpoint.RequireCapability(DataProtectionCapabilities.KekSelfTest));
// POST .../self-test?provider=AzureKeyVault  →  { provider, ok, kekIdFingerprint, latencyMs, errorKind? }

UseRateLimiter() goes after UseAuthentication(). The limiter partitions by caller — User.Identity.Name, else the remote IP. Before authentication has run there is no user, so every caller falls into its IP's bucket, and behind a proxy that is one bucket for every operator.

DataProtectionCapabilities.KekSelfTest ("dataprotection:kek:self_test") is a library-owned string constant — mirror it into your capability catalog and grant it to your admin/super-admin role. The response returns a fingerprint of the kek_id (never the vault URI) and a classified errorKind (never raw provider exception text). The self-test calls the adapter directly — it writes nothing to the keyring.

Operating the key lifecycle — every lever, and how to smoke-test it

Everything below is opt-in. Wire the endpoints you want; a surface you do not map does not exist. Every endpoint takes a required configure callback where you apply your own capability gate — this package depends on no auth middleware.

The full surface

Automatic, once AddKeyringMaintenance is wired: bootstrap the first DEK, rotate at MaxDekAge, and re-wrap onto the current KEK — all leader-elected. The manual levers exist for the cases a timer cannot serve: a suspected exposure, a rehearsal, a cutover you want logged and watched.

Smoke-testing the whole lifecycle locally

Run this against a local instance with Local custody — no vault needed. It exercises both lifecycles end to end; each step has an observable result, so a failure tells you which mechanism broke.

BASE=http://localhost:5000/api/v3/admin/dataprotection
AUTH="Authorization: Bearer $TOKEN"          # a principal holding the four capabilities above

# 1. KEK reachable? (no keyring writes, no real data)
curl -sX POST "$BASE/self-test?provider=Local" -H "$AUTH"     # → { ok: true, kekIdFingerprint, latencyMs }

# 2. Keyring bootstrapped at boot: exactly one active DEK, everything on the current KEK
curl -s "$BASE/kek-status" -H "$AUTH"                          # → stale: 0, byKek: [ { isCurrent: true, … } ]

# 3. Encrypt something through the app, then read the column: it must start `hxdp:v1.<key_id>.`
#    — that key_id is the generation, in the clear, by design.

# 4. Rotate the DEK. New generation; the previous is RETIRED and retained, so step-3 data still decrypts.
curl -sX POST "$BASE/dek/rotate" -H "$AUTH"                    # → { keyId: "<new>" }
#    Re-read the step-3 row (still decrypts, still its old key_id) and write a new row (new key_id).

# 5. Which tables still sit on the old generation?
curl -s "$BASE/dek-status" -H "$AUTH"

Then the KEK half — swap the local master, which is the part with a real failure mode:

# 6. Set DATA_PROTECTION_MASTER_KEY_BASE64_OLD=<current>, DATA_PROTECTION_MASTER_KEY_BASE64=<new>. Restart.
curl -s "$BASE/kek-status" -H "$AUTH"     # → every DEK still readable, now reported against TWO kek_ids

# 7. Let maintenance run (or call ReWrapAllAsync). Re-check: one kek_id, stale 0.
# 8. Prove the old master is unused BEFORE deleting it:
#    AssertNoneWrappedByAsync("LocalPrevious") — throws if any DEK still needs it.
# 9. Only now remove DATA_PROTECTION_MASTER_KEY_BASE64_OLD and restart.

Step 8 is the one step you cannot skip: dropping a master while a live DEK is still wrapped by it is unrecoverable. The gate is fail-closed for that reason.

What this does not cover. Destruction is deliberately slow and cannot be smoke-tested in one sitting — a retired DEK waits out a quarantine window, then a recoverable soft-delete window, with a fresh drain-veto re-checked at the shred instant. Rehearse it with shortened windows in a scratch environment, never by shortening them in production.

Keep this as a checklist you run by hand against a local instance. It is not a CI job: it asserts operator-visible behaviour of a running deployment, and a green run in a container proves nothing about a cluster.

Operations

KEK-retention, compromise/rotation, encrypted-column rules, and the KV rotation burst budget are in the ops runbook: docs/data-protection/OPS_RUNBOOK.md. The KEK-retention rule is data-loss-class — read it before rotating a KEK in production.