hexasync.access.client

The client for the HexaSync graph authorization engine. Five things live here, and only the last two need a guide:

GraphAuthorizationClient	ask the engine a permission question
GraphDefinitionsClient	read the model — entity types, relations, permissions
GraphRegistrationClient	register this service as an entity authority
Outbox/	publish this service's own changes to the engine, transactionally
Projection/	hold a local copy of the engine's answers and read them without a round trip

Both of the last two put tables in your database, and each has its own ModelBuilder call — see Put the tables in your model. They are independent: publishing without reading and reading without publishing are both ordinary.

---

The outbox, in one paragraph

Your service owns some entities — profiles, tenants, devices. The engine needs to know they exist and what they are called, and it must learn this exactly when your write commits and never otherwise. So you write a row into graph_outbox inside your own transaction, and a background driver walks that table in commit order and delivers each entry to the engine. If your transaction rolls back, the entry was never there. If delivery fails, the entry is still there.

One cursor, in global commit order. The walk reads every lane in one (xid, id)-ordered pass — not a cursor per lane — so an entry committed before another is delivered before it whatever lanes they are on. That is what makes it safe for the engine's metadata upsert to carry no monotonic guard.

Two lanes ship with the package:

• graph.init — this resource now exists, with these initial relations. Delivered to POST /api/v3/graph/resources/init.
• graph.metadata — this entity's label changed. Delivered to PUT /api/v3/graph/entities/{type}/{id}/metadata.

---

Wiring it up — four steps, and none of them is optional

1. Put the tables in your model

The package owns the shape; you own the migration. There are two model extensions, and you call the one(s) whose feature you use:

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    base.OnModelCreating(modelBuilder);

    modelBuilder.ApplyGraphOutboxModel();   // graph_outbox, graph_outbox_state, graph_outbox_dead
    modelBuilder.ApplyProjectionModel();    // projection_header, projection_detail  -- only if you read projections
}

call it when	tables
ApplyGraphOutboxModel()	you publish graph changes — i.e. you registered the driver in step 2	graph_outbox, graph_outbox_state, graph_outbox_dead
ApplyProjectionModel()	you read projections locally — ProjectionReader, ProjectionLocalReader, ProjectionSweepService	projection_header, projection_detail

They are independent: publishing without reading and reading without publishing are both ordinary. Calling one you do not use costs you two empty tables in your migration, so leave it out.

⚠️ ApplyProjectionModel creates the two tables UNLOGGED, and that is deliberate. A projection is a cache of someone else's authoritative data — losing it on an unclean shutdown means re-fetching, not losing anything — and unlogged tables skip the WAL for a table that is rewritten wholesale on every refresh. It also means they are empty after a crash restart, so nothing may treat their contents as durable.

Then scaffold your own migration as you would for any other change:

dotnet ef migrations add GraphOutbox --project <your-ef-project>

Why not a DDL script the package hands you? It used to be exactly that, and it rotted in a way worth understanding before you reach for the same shortcut elsewhere. CREATE TABLE IF NOT EXISTS does nothing to a database that already has the table, so every column added after the first release had to arrive as an appended ALTER TABLE … ADD COLUMN IF NOT EXISTS. That list only grows, and it is applied by whatever you remember to run. The failure that actually happened is worse than untidy: a consumer ran the script as raw SQL from inside a migration, so the tables existed in its database and nothing at all was in its model snapshot — and the next migrations add could not see them.

An IEntityTypeConfiguration describes the shape as it is, not as a history of edits. Your migrations add computes the diff, and your snapshot carries the tables, so the migration after this one is computed against the truth.

⚠️ ApplyConfigurationsFromAssembly on your own assembly will not find these — they live in this one. That is why the call is explicit.

2. Register the driver

services.AddGraphOutbox(o =>
{
    o.DeliveryUrl = config["GRAPH_ENGINE_URL"];          // required, checked at boot
    o.NotificationUrl = config["ALERT_WEBHOOK_URL"];     // optional today
    o.Notify = (notice, ct) => alerts.SendAsync(notice, ct);   // required, see below
});

Two things must already be in your DI container, because both are deployment-wide decisions and neither is the package's to make:

• NpgsqlDataSource — the outbox tables live in your database. That is the entire point of an outbox.
• IDistributedLockProvider — the leader lock. Its backend is yours to choose; this workspace uses Redis via Medallion.

AddGraphOutbox registers the driver, both standard processors, the named HttpClient, and the boot-time URL check. It resolves nothing at registration time.

3. Attach your service identity to the delivery client

The package sets the address. You supply the credential — every consumer reaches the engine as itself, and a library that picked a token source would be choosing your infrastructure:

services.AddHttpClient(GraphOutboxHttp.ClientName)
        .AddHttpMessageHandler<InnerServiceTokenHandler>();

Skip this and every entry parks on a 401.

4. Enqueue inside your own transaction

await using var tx = await connection.BeginTransactionAsync(ct);

// … your mutation, FIRST …
await UpdateTheProfileRow(connection, tx, profile, ct);

// … then the enqueue, on the SAME connection and transaction …
await GraphOutboxWriter.EnqueueAsync(
    connection, tx,
    Guid.CreateVersion7(),
    GraphOutboxLanes.Init,
    JsonSerializer.Serialize(new
    {
        resourceType = "profile",
        resourceId = profile.Id.ToString("D"),
        relations = new[] { new { relation = "parent", subjectType = "user", subjectId = ownerId.ToString("D") } },
        requestedBy = ownerId.ToString("D"),
    }),
    ct);

await tx.CommitAsync(CancellationToken.None);

🔴 Enqueue AFTER the row update, never before. The row lock is what orders two concurrent mutations of the same entity: the second cannot reach its enqueue until the first commits, so ids come out in commit order. Insert first and they invert — the transaction that has not taken a lock yet gets the lower id, and the later change is delivered as if it were earlier.

🔴 The id is yours to supply — normally Guid.CreateVersion7(). There is no column default, because uuidv7() is a PostgreSQL 18 built-in and this package does not control your server version.

---

The shape of the running system

your handler ──┬── UPDATE profiles          ┐
               └── INSERT graph_outbox      ┘ one transaction

                        ▼

  GraphOutboxService  (one per pod; exactly one holds `graph-outbox-driver`)
        │
        └─ ScanOnceAsync(LIMIT 100)      ONE cursor, every lane, in (xid, id) commit order
             │
             │  the batch is cut into CONTIGUOUS RUNS of one lane
             ├─ run of 12 × graph.init ─────→ GraphOutboxInitProcessor ─────→ engine
             ├─ run of  3 × graph.metadata ─→ GraphOutboxMetadataProcessor ─→ engine
             └─ run of  1 × graph.init ─────→ GraphOutboxInitProcessor ─────→ engine


  GraphOutboxRetentionService  (one per pod; exactly one holds `…-retention`)
        │
        └─ every RetentionInterval: two bounded DELETEs, at or BELOW the cursor

🟢 The two services are independent, and a DIFFERENT pod may hold each lease. Nothing coordinates them and nothing needs to: retention only ever touches rows at or below the cursor, and the walk only ever reads rows above it, so the two sets are disjoint by construction.

One driver, ONE cursor, every lane. The lanes are the registered processors, not a SELECT DISTINCT(lane). A pass is one bounded SELECT over the whole table in commit order — BatchSize entries whatever happens inside it — cut into runs of consecutive same-lane entries and handed to each lane's processor in turn.

🔴 Isolation between lanes is GONE since 1.16, and that is the trade. A retryable failure now blocks everything behind it rather than one lane — a 401 retries forever, so it stalls the whole outbox. Permanent failures still park and advance, so the exposure is narrower than it first looks. This was chosen because the alternative is worse and silent: per-lane cursors preserve commit order inside a lane and discard it between lanes, so once a delete lane exists a delete can overtake a metadata write and the row it deleted comes back. Nothing errors and nothing logs. A blocked queue is visible delay; resurrection is silent corruption.

One leader. Every pod starts the driver; one acquires the lock and works, the rest stand by and take over on eviction. Standby is the normal state, not an error. If the leader loses its lease mid-pass it stops immediately — a holder that kept working after eviction would produce the exact double-driver state the lock exists to prevent.

🔴 THE LOCK IS THE SINGLE-WRITER GUARANTEE, and YOU own the lease TTL. This paragraph used to say the opposite — that the lock bought "waste, not correctness" and that a conditional claim in SQL was what stopped two pods notifying. That second guard was removed by owner ruling on 2026-09-03: only the driver calls the notifier, it runs under the lock, and a lease long enough to outlast any real pause makes a second concurrent caller a case not worth carrying code for.

What that puts on you, because this package cannot do it: it takes DistributedLock.Core — the abstraction only — so the expiry is set where you construct your provider. Configure a lease that comfortably outlasts your worst pause, and remember it is the same knob as failover: a leader that is OOM-killed leaves the outbox undrained until its lease expires.

🟢 Still true and still load-bearing: the walk tolerates double delivery against an idempotent destination. Do not remove that on the strength of the lock.

---

Adding your own lane

Implement the interface and register it. Nothing else changes:

public sealed class DeviceProjectionProcessor : IGraphOutboxLaneProcessor
{
    public string Lane => "myservice.devices";

    public async Task<IReadOnlyList<DeliveryOutcome>> DeliverAsync(
        IReadOnlyList<GraphOutboxEntry> run, CancellationToken ct)
    {
        var outcomes = new DeliveryOutcome[run.Count];

        for (var i = 0; i < run.Count; i++)                      // IN ORDER. One at a time.
        {
            var response = await _client.PostAsync("…", Body(run[i].Payload), ct);
            outcomes[i] = DeliveryOutcome.FromStatus((int)response.StatusCode);

            if (!outcomes[i].IsDelivered && !outcomes[i].ShouldPark)
            {
                // not attempted -- nothing was decided about these, so they are retries
                for (var rest = i + 1; rest < run.Count; rest++)
                    outcomes[rest] = DeliveryOutcome.Retry("not attempted");
                break;
            }
        }

        return outcomes;
    }
}

services.AddScoped<IGraphOutboxLaneProcessor, DeviceProjectionProcessor>();

🔴 DELIVER THE RUN IN ORDER, ONE AT A TIME. This is the one rule the batch shape hands to you, and nothing in the type system enforces it. await Task.WhenAll(run.Select(Send)) looks like an obvious throughput win and it reproduces, inside a single run, the out-of-order delivery the single cursor exists to remove. If a run ever has to be pipelined, the partition must be the entity, not the operation.

🔴 Exactly one outcome per entry, in the same order. An entry you chose not to attempt is a Retry — that is what not attempted means. A count that does not match the run fails the whole run and is logged as a processor defect; it is never read as "the rest succeeded", which would advance the cursor past entries nobody classified.

🟢 Why a list and not one status for the run. One status cannot say "9 delivered, 1 parked, 2 not attempted" — and being able to say that is what keeps stop at the first failure, never skip and who advances the cursor inside the scanner, where they are written once, instead of in every processor you ever write.

Return, don't throw, for an expected refusal. A throw is caught and treated as a retry, which is the right default for a broken transport — it keeps deliverable data out of the dead-letter table.

---

What happens when delivery fails

Response	What the driver does
2xx	delivered; the cursor advances past it
permanent (the 400/409 allowlist)	the entry moves to graph_outbox_dead and the lane continues
anything else	retried forever, and the lane does not advance past it

There is no attempt limit, and that is the design: what matters is whether the response can ever change, not how many times you have asked.

A parked entry is logged at Warning every pass, and raises a notice through your Notify delegate. That delegate is required for a reason — an earlier version halted the lane on a permanent failure, which was impossible to ignore; parking is easy to ignore, and a notice nobody configured is the same silence in a different place.

Triage and replay through GraphOutboxDeadLetterStore — the list route deliberately carries no payloads, and a requeue inserts a new entry with a fresh id so both attempts stay traceable.

What to alert on

The stuck notice, and nothing else. first_stuck_at is stamped when a round ends in a retry and cleared by any round that delivered or found nothing, so it reports how long the outbox has been failing and an idle outbox cannot raise it. It reaches you through your Notify delegate after StuckAfter (2 min), throttled to one per 15 minutes, carrying the last error and the failing entry's lane — the state is global since 1.16, so the lane travels in the notice rather than on the cursor.

🔴 Do not build a lag metric from the walk's cursor. The package shipped one and removed it in story 1.15, because it is wrong in both directions. An idle lane's cursor is frozen while the horizon advances with every unrelated transaction in your database, so the number climbs forever on a healthy system. And when the horizon genuinely is pinned by a long-running query, the lane drains everything below it and the number falls to ~0 — it reads healthiest exactly when the problem is real. A pinned horizon is a database condition with its own monitoring; this package watches the queue.

The walk still cannot advance past the oldest in-flight transaction anywhere in your database — that cost is real, it is simply not observable from here.

---

Reading projections locally

A projection is a local copy of the engine's answers for one subject: which resources they can touch, and with which bits. You refresh it on a schedule or on a signal, then answer permission questions from your own database instead of over the network.

Two tables, and ApplyProjectionModel() is what puts them in your migration:

projection_header	one row per generation — a complete, self-consistent snapshot. Exactly one is active
projection_detail	one row per resource in that generation, carrying the 64-bit mask

A refresh writes a whole new generation and flips active at the end, so a reader never sees a half-built one. The superseded generation is then dead weight, which is what the sweep is for:

services.AddSingleton(new ProjectionSweepOptions { DataSource = dataSource });
services.AddHostedService<ProjectionSweepService>();

🟠 ProjectionSweepService deliberately takes NO leader lock, and that is not an oversight — its statements are idempotent, each filtered by a partial index and empty in the common case, so running it on every pod is the same as running it on one. Its failures cost storage, not correctness. (Contrast the outbox's retention sweep, which is bounded and therefore does take one — see Options.)

🔴 Both tables are UNLOGGED. They are a cache of someone else's authoritative data, they are rewritten wholesale on every refresh, and skipping the WAL is most of the point. They come back EMPTY after an unclean shutdown — so treat an absent projection as "refresh it", never as "this subject has no access".

---

Options

	default
DeliveryUrl	—	required; the engine's base address, checked at boot
NotificationUrl	—	where notices go; optional today
Notify	—	required; the channel is yours
BatchSize	100	entries per pass, across every lane
IdleDelay	5s	after a pass that delivered nothing
ErrorDelay	30s	after a pass that threw — a different thing from finding nothing
LeaderLockName	graph-outbox-driver	deployment-scoped, not per lane
LeaderWait	30s	how long one acquisition attempt blocks; standby retries forever
UseStandardProcessors	true	set false only to substitute your own; an extra lane is added, not swapped
RetentionWindow	7d	how long a delivered entry stays readable before it is swept
RetentionInterval	1h	how often the retention loop sweeps
RetentionBatchSize	1000	rows removed per pass per table, so a first run against a backlog stays bounded
RetentionDisabled	false	when true the retention service is not registered at all
RetentionLockName	{LeaderLockName}-retention	its own lease — see below for why it must not be the driver's

🔴 Retention runs in its own hosted service with its own leader lock, registered for you by AddGraphOutbox. Two things about that are worth knowing before you change either name:

• It must not share LeaderLockName. The driver holds that lease continuously while it is the leader, so a second service waiting on it would acquire only when the driver stopped — retention would simply never run, with no error and no log.
• The default is derived rather than constant. Two applications sharing one lock backend must not fight; if you renamed LeaderLockName to avoid exactly that, you get a distinct retention lock for free. A fixed literal would have given you one retention leader globally and left the second application's table unswept, silently.

Why a lock here at all, when ProjectionSweepService deliberately has none? Because this sweep is bounded. The projection sweep is unbounded and convergent — running it N times equals running it once, so a lock would buy only saved round trips. Retention removes at most RetentionBatchSize per pass, so N unsynchronised pods remove up to N× the ceiling that option promises — and that ceiling exists for the first run against a backlog, which is exactly when it matters.

---

Upgrading to 1.18 + 1.20 — one migration, three breaking changes

These two ship together. Scaffold ONE migration after taking the package, not two:

dotnet ef migrations add GraphOutboxState --project <your-ef-project>

1. graph_outbox_cursor is DROPPED and replaced by graph_outbox_state. It was one row of typed columns; it is now (key text primary key, value text null). The nine facts keep their names as keys — last_scanned_xid, last_scanned_id, last_scan_at, failed_attempts, last_error, last_dead_letter_at, last_dead_letter_notification_at, first_stuck_at, last_stuck_notification_at.

Why. Every fact this package tracked about itself was a column, so adding one cost you a migration — three releases in a row did exactly that, and a fourth wanted a housekeeping timestamp and declined to add it rather than make you pay again. A new fact is now an INSERT by the package and nothing at all by you. It also retires a column named lane that had held the constant * since 1.16 and had not meant a lane since.

🔴 Your development database re-delivers everything once, for the same reason and with the same consequences as the 1.16 note below: the new table starts empty and the walk starts at zero. Harmless — the destination is idempotent — but do not be surprised by it. There is no data migration; nothing carries the old row forward.

🟠 If you have not yet scaffolded 1.16's index re-cut, both land in this one migration. Then 1.16's checklist item — "a DropIndex and a CreateIndex, and nothing else" — no longer holds: you will also see the cursor table dropped and the state table created. That is correct; do not reject it.

2. GraphOutboxSchema is GONE. Both of its public consts go with it — CursorLane (there is no keyed cursor row to name) and SeedCursorSql (there is no seed: reads coalesce an absent key to its default and every write is an upsert). If you referenced either, GraphOutboxState's key constants are the replacement.

3. GET …/lanes always answers 200; the 404 arm is deleted. The endpoint's result type changes from Results<Ok<GraphOutboxHealth>, NotFound> to Ok<GraphOutboxHealth> — source-breaking if you matched on the union. The response body is unchanged. 404 used to mean "nothing has ever scanned", which was only ever safe because the old cursor row was seeded on every pass; without a seed a healthy-but-idle outbox writes nothing at all, so keeping the arm would have answered 404 for ever on a perfectly working deployment.

🟢 Not breaking, but new: retention moved out of the delivery pass into its own hosted service — AddGraphOutbox registers it, so there is nothing for you to wire. See Options for its lock.

---

Upgrading to 1.16 — three breaking changes

1. IGraphOutboxLaneProcessor.DeliverAsync takes a RUN and returns one outcome per entry. Every processor stops compiling, which is the intent — see Adding your own lane above for the shape, and the ordering rule that comes with it.

2. GET …/lanes answers with ONE object, and ALWAYS 200. The scan and notification stamps live in graph_outbox_state and cover every lane. A client reading .lanes[0] gets nothing rather than an error. 🔴 Since 1.18 the 404 arm is gone: package state has no seeded row, so a healthy-but-idle outbox has simply written nothing and every stamp is null. The response record is unchanged; only the endpoint's result type is.

3. A LOCAL database re-delivers everything once. The cursor is keyed on a constant instead of a lane name, so an existing development database keeps its old per-lane rows — nothing reads them — and the new row starts at zero. Harmless: the destination is idempotent and re-delivery is the failure mode this design is built for. There is no data migration and no production deployment holds cursor rows.

🟠 The migration is a schema change only, and it is the usual one: migrations add from the model. It drops ix_graph_outbox_walk and recreates it on (xid, id). Do not skip it — the walk no longer filters on kind, so against the old (kind, xid, id) index the keyset has an unbound leading column and every pass degrades to a sequential scan and a sort of the whole table. Probed against postgres:18, 60 000 rows.

---

Two failure modes worth recognising

"Everything is healthy and the graph is stale." The driver catches every exception a pass throws, logs it, and sleeps. So a misconfiguration produced an outbox that looked alive and delivered nothing — which is why the URLs are checked at boot and the host refuses to start without them.

An entry whose lane has no registered processor is PARKED, immediately. It cannot become deliverable by retrying — only by a deployment — and under one cursor a retry would block every entry behind it. The payload survives in graph_outbox_dead and POST /{id}/requeue replays it once the processor ships. 🟠 So a producer deployed before its processor sends its whole stream to the dead table, and requeue is one row at a time: a bulk route is deliberately not in the package, because the recovery scale is bounded by how long a producer runs without its processor. The driver refuses to scan at all while no processors are registered, which is what keeps that case recoverable.

"Registered and idle" looks exactly like "not registered at all." A consumer enqueued onto both lanes for a month while nothing drained them, because wiring the outbox took six registrations per lane and nobody did it. That is what AddGraphOutbox exists to prevent — and why the driver logs a Warning, every pass, when no processors are registered.