The problem

Account statements are among the most sensitive documents a bank holds, and regulation requires them to be kept for seven years. This platform generates, stores and delivers them at a design scale of about 30 million a month.

A reference implementation of a bank statement platform, built to work through the hardest constraints in the domain end to end. It isn't deployed; the repository runs locally with one command.

The central tension is not throughput. Statements are stored under S3 Object Lock in Compliance mode, so for seven years nobody can delete them: not an administrator, not the root account. But POPIA gives a customer the right to erasure. Inside the retention window, deletion isn't difficult. It is impossible.

The answer, as a chain of consequences

  1. Deletion is impossible, so erasure has to be cryptographic. Destroy the customer's key and every copy of their statements, live, versioned, backed up and replicated, becomes permanently unreadable while the ciphertext stays exactly where the law requires it.
  2. That requires client-side envelope encryption under a per-customer key. Storage-managed encryption can't erase anything, because the mapping from object to key belongs to the storage service, not to us.
  3. One cloud KMS key per customer would cost 26 million dollars a month at 26 million customers. So a three-tier hierarchy: 1,024 cohort keys in KMS wrap one key per customer in the database, which wrap one key per object. Erasure stays per-customer, and the KMS line item drops to about 1,024 dollars a month. The trade-off is stated in the ADR: a customer key is now protected by the database's access controls rather than an HSM.
  4. Because objects are encrypted client-side, delivery can't hand out presigned storage URLs. Everything is decrypted and streamed through a gateway. The cost model shows that's cheap here anyway: peak delivery is about 6 MB/s, which comes to roughly 22 dollars a month of egress.
  5. .NET's AesGcm refuses to stream, and for good reason: a naive streaming decryptor hands out plaintext before it has checked the tag. So statements are encrypted as a framed AEAD format, 64 KiB frames with per-frame tags bound to the statement's identity, and the gateway authenticates each frame before it leaves. A test downloads a 200 MB encrypted statement through the real gateway in constant memory.

One test ties the chain together. It encrypts a statement through the real key stack, destroys the customer key, asserts the ciphertext is still there (nothing deleted it, and under a Compliance lock nothing could), and then asserts that no fresh key cache can unwrap it and the erased customer cannot be given a new key.

Key engineering decisions

Two subsystems with opposite scaling profiles

Generation is a monthly burst of about 1,400 renders a second, scaling from zero to 400 workers for a few hours. Delivery is a steady trickle, about half a request a second with 30 at peak, that has to answer in under a second and is visible to a person when it fails. Coupling them means the customer API is sized for month-end all month. That, and not fashion, is why this is four deployables rather than a modular monolith: an authenticated API, an unauthenticated download gateway, a generation worker, and a retention worker that is the only process holding delete rights. The ADR says it plainly: for most systems this size a modular monolith is correct, and the divergent scaling profile alone earns the split.

Single-use links with nothing to learn from a failure

Issuing and redeeming a single-use download linkThe customer asks Delivery.Api for a download link; the API stores only a hash of the token and returns the plaintext once. The customer redeems it at Download.Gateway, which consumes the token atomically before streaming the decrypted PDF frame by frame from object storage. A second request with the same link gets a uniform 404; the reason is recorded only in the audit trail.CustomerDelivery.ApiDownload.GatewayPostgreSQLObject storagePOST /statements/{id}/download-links (JWT)insert SHA-256(token) + audit LINK_ISSUED, one transaction201 · URL with the plaintext token (exists nowhere else)GET /v1/d/{token}atomic consume UPDATE … RETURNING + audit DOWNLOAD_STARTEDexactly one concurrent redeemer winsGET ciphertext, streamed200 · PDF, decrypted and tag-verified frame by framethe same link again404 · uniform; the real reason lives only in the audittrail
Issuing and redeeming a single-use download linkThe customer asks Delivery.Api for a download link; the API stores only a hash of the token and returns the plaintext once. The customer redeems it at Download.Gateway, which consumes the token atomically before streaming the decrypted PDF frame by frame from object storage. A second request with the same link gets a uniform 404; the reason is recorded only in the audit trail.CustomerAPIGatewayPostgresStoragePOST/statements/{id}/download-links(JWT)insert SHA-256(token) +audit LINK_ISSUED, onetransaction201 · URL with theplaintext token (existsnowhere else)GET /v1/d/{token}atomic consume UPDATE …RETURNING + auditDOWNLOAD_STARTEDexactly one concurrentredeemer winsGET ciphertext, streamed200 · PDF, decrypted andtag-verified frame by framethe same link again404 · uniform; the realreason lives only in theaudit trail
Issuing and redeeming a link. The token is consumed in one atomic statement, with its audit event, before a byte is streamed. Redrawn from the repository README.

Tokens are 256 random bits, stored only as a SHA-256 hash, bound to one customer and one statement. Redemption is a single UPDATE … RETURNING that consumes the token and writes the audit event in the same transaction, before the content stream opens, so exactly one concurrent redeemer wins. The token stays consumed if the transfer fails; releasing it on abort would let an attacker replay it by killing the connection. Expired, revoked, consumed and never-existed links return the same 404, same body, same headers, padded to the same timing floor, and an integration test asserts the four responses are byte-identical. The trade-off: a customer whose download drops has to ask for a new link.

Legal conflicts are surfaced, never resolved in code

A pure decision engine in the domain project encodes the precedence: legal hold, then Object Lock, then statutory retention. A hold can extend retention and nothing can shorten it. Every refusal names its basis: an erasure request blocked by retention gets a 409 that cites the statute and the date the obligation ends, and one blocked by a hold cites the case reference. The test's own assertion message says why: a bare 409 is a bug report, not a defensible response.

Architecture as executable rules

The domain project references nothing, and a test proves it. Query discipline is enforced the same way: no OFFSET pagination, an explicit timeout on every command, no SELECT *. The API and gateway cannot reference any crypto type, persistence handles wrapped keys only, and money is never a double anywhere. A separate integration test walks every endpoint and fails unless it is authorised or on a justified allow-list: the test that catches the endpoint someone adds next year. Storage listing is a documented convention rather than a test, with one bounded exception in the orphan sweep.

PgBouncer because of arithmetic

Four hundred generation workers with a pool of five each ask for 2,000 database connections, 2,210 across the fleet, against a server whose practical ceiling is a few hundred. Transaction pooling turns that into about 100 backends. The repository documents what breaks silently behind it: advisory locks, connect-time SET, LISTEN/NOTIFY, held cursors, temp tables across statements. Leader election therefore uses a lease table with a fence token, not the usual advisory-lock recipe.

How to explore it

One command: cp .env.example .env && docker compose up --build. Fourteen containers start; a migrator applies 24 forward-only migrations and gates everything behind it, and a one-shot job seeds demo data and drives one real generation run.

The README's five-minute walkthrough is the most vivid demonstration of the design. You get a token, list a customer's statements, issue a single-use link and download the PDF. Then you paste the same link again and get a 404. Both attempts are in the tamper-evident audit trail, newest first: the denial with its real reason, the completed download, the issued link.

Where to read:

On scale: the design targets are labelled as requirements, not measurements. What was measured, on a four-core machine against a stack seeded to 85.7 million statements, is written down with its provenance: the link-issue path saturates at about 263 issues a second with the WAL flush as the ceiling, and a partition-pruned statement query runs in 7.5 ms at that volume. Generation throughput and the 1,400 renders a second target are explicitly unverified.

What it deliberately doesn't do

  • No HTTP range requests. They can't be reconciled with single-use tokens, and the tokens won.
  • No automatic orphan deletion. The sweep reports and never deletes: a Compliance lock forbids it anyway, and inventory-driven deletion turns a comparison bug into data loss.
  • KMS key rotation is not built. Every object records which cohort key wrapped it, so rotation never rewrites history, but the rotation job itself is future work.
  • The message broker transport is a logging sink. The transactional outbox, relay and at-least-once semantics are real; the sink is a structured log until a consumer exists.
  • Multi-region is designed in the scale document's "where it breaks" section and not built; single-region is stated as a limit.
  • Locally, KMS is a development key provider that refuses to start outside Development. The AWS KMS provider is implemented with gated tests but has never run against real AWS KMS from this project.

And the one that matters most: audit-chain heads sit in the same database as the events they attest. A privileged insider who can rewrite both can forge a self-consistent chain, and truncation from the tail is invisible from inside. External anchoring is the fix; the seam exists and is a no-op today. It is the one open row in the threat model, named rather than hidden.

Stack

.NET 10 and C#, ASP.NET Core minimal hosting with a shared service-defaults project across the four services. PostgreSQL 17 range-partitioned by month, behind PgBouncer in transaction mode, with Dapper and DbUp forward-only migrations rather than Entity Framework. S3 Object Lock for storage, MinIO locally. Redis for rate limiting. OpenTelemetry traces, metrics and logs to the Aspire dashboard. UUIDv7 primary keys. xUnit v3 with Testcontainers, so integration tests run against real PostgreSQL and MinIO; 689 tests, 163 of which skip with a stated reason when there is no container runtime. Chiseled, non-root runtime images. Scalar for API docs. A six-job GitHub Actions pipeline that boots the full stack, checks non-root containers and leader election, and scans every image.