Payment Platform
Safely orchestrate provider payments, idempotency, ledger entries, webhooks, and reconciliation.
Product overview
Teach safe payment orchestration around a regulated provider: payment intents, tokenized methods, idempotent commands, append-only ledger concepts, verified webhooks, and reconciliation.
Main actors
Merchant, Customer, Finance operator, Risk analyst.
Assumptions
- A compliant payment service provider collects and tokenizes card details
- The platform stores provider tokens, never raw card data or CVV
- The illustrative ledger is not a bank ledger
- Production use requires legal, security, PCI, tax, and jurisdiction-specific review
Non-goals
- Processing card numbers directly
- Holding deposits as a bank
- Giving compliance certification advice
- Implementing card-network protocols
Requirements
Must-have
- Tenant-scoped payment intents
- Provider token orchestration
- Idempotent confirm/refund
- Verified webhook inbox
- Balanced append-only ledger and reconciliation
Should-have
- Partial capture/refund
- Provider adapter
- Disputes projection
- Operational exceptions queue
Future scope
- Multiple providers
- Payout orchestration
- Risk rules
- Multi-currency conversion
Non-functional decisions
| Concern | Decision |
|---|---|
| Correctness | Money uses integer minor units and ISO currency; every mutation has durable idempotency and state preconditions. |
| Durability | Ledger postings and resulting domain transition commit atomically; provider state is reconciled. |
| Availability | Ambiguous provider outcomes become pending, never guessed success/failure. |
| Security | Card entry stays on provider-hosted fields; secrets are vault-managed, scoped, and rotated. |
| Auditability | Commands, webhooks, ledger entries, overrides, and reconciliation outcomes are immutable/audited. |
User roles
| Role | Access boundary |
|---|---|
| Merchant service | Own tenant intents/refunds using scoped API credential |
| Finance operator | Tenant reports and controlled exception resolution |
| Risk analyst | Risk signals/holds; no secret or raw instrument access |
| Platform operator | Provider health and replay controls with dual control |
Core workflows
Create and confirm payment
Merchant creates intent with idempotency key → Server records amount/currency/status → Client uses provider-hosted collection → Merchant confirms intent → Provider may require customer action → Webhook/reconciliation determines authoritative provider outcome → Post balanced internal ledger entries
Refund
Lock payment/refundable aggregate → Validate amount and merchant → Create pending refund → Call provider using stable reference → Apply verified webhook/result → Post reversal entries → Reconcile
Domain model
| Entities | Relationship |
|---|---|
Merchant → PaymentIntent → PaymentAttempt | Intent is merchant goal; attempts record provider interactions |
PaymentIntent → Capture / Refund | Amount-limited state transitions |
LedgerTransaction → LedgerEntry | At least two entries sum to zero per currency |
WebhookInbox | Verified provider event id and processing state |
IdempotencyRecord | Merchant/key binds request hash to stored response |
ReconciliationException | Explicit mismatch workflow |
Database design
PostgreSQL remains the transactional source of truth. Add specialized stores only for the workloads called out below.
payment_intents — Merchant payment aggregate
payment_intents
---------------
id UUID PK
merchant_id UUID FK
merchant_reference TEXT
amount_minor BIGINT
currency CHAR(3)
status TEXT
provider_ref TEXT NULL
version INT
Constraints: UNIQUE(merchant_id, merchant_reference); amount_minor > 0
Indexes: (merchant_id, created_at DESC); (status, updated_at)
idempotency_records — Replay-safe command result
idempotency_records
-------------------
merchant_id UUID
key TEXT
request_hash TEXT
status TEXT
response_code INT
response_body JSONB
expires_at TIMESTAMPTZ
Constraints: PK(merchant_id, key)
Indexes: (expires_at)
ledger_transactions — Posting envelope
ledger_transactions
-------------------
id UUID PK
merchant_id UUID
source_type TEXT
source_id UUID
currency CHAR(3)
created_at TIMESTAMPTZ
Constraints: UNIQUE(merchant_id, source_type, source_id)
Indexes: (merchant_id, created_at, id)
ledger_entries — Immutable debit/credit amount
ledger_entries
--------------
id UUID PK
transaction_id UUID FK
account_id UUID FK
amount_minor BIGINT
currency CHAR(3)
Constraints: amount_minor <> 0; deferred validation: sum per transaction/currency = 0; no update/delete
Indexes: (account_id, id)
webhook_inbox — Verified provider message receipt
webhook_inbox
-------------
provider TEXT
provider_event_id TEXT
received_at TIMESTAMPTZ
payload_ref TEXT
status TEXT
attempts INT
Constraints: PK(provider, provider_event_id)
Indexes: (status, received_at)
Transaction and retention boundaries
- Reserve idempotency key/request hash before side effect; concurrent identical call waits or returns in-progress
- Apply valid payment transition and balanced ledger transaction atomically
- Refund creation locks payment aggregate and sums completed/pending refunds
- Never edit ledger entries; correct with linked reversal
- External provider call is outside DB transaction and uses stable provider idempotency reference
API design
POST /v1/payment-intents
POST /v1/payment-intents/:id/confirm
GET /v1/payment-intents/:id
POST /v1/refunds
POST /v1/webhooks/provider
GET /v1/balance-transactions?cursor=…
| Endpoint | Purpose and permission | Idempotency / status |
|---|---|---|
POST /v1/payment-intents | Create merchant payment intent Merchant credential with payments:write | Required; merchant scoped and payload hashed 201, 409, 422 |
POST /v1/payment-intents/:id/confirm | Begin/continue provider confirmation Owning merchant/customer secret boundary | Required 200, 202, 402, 409 |
GET /v1/payment-intents/:id | Read platform state Owning merchant or restricted client secret view | Not required 200, 403, 404 |
POST /v1/refunds | Request full/partial refund Merchant refunds:write | Required 201, 202, 409, 422 |
POST /v1/webhooks/provider | Receive provider event Verified signature/timestamp; not user auth | Provider event id unique 200, 400 |
GET /v1/balance-transactions?cursor=… | Keyset ledger view Merchant finance read | Not required 200, 403 |
List endpoints use opaque keyset cursors unless a small bounded collection is explicit. Unknown objects return 404 where 403 would leak existence.
Architecture
System context
Rendering diagram…
Main application architecture
Rendering diagram…
Critical workflow sequence
Rendering diagram…
Event model
| Event | Producer → consumers | Delivery |
|---|---|---|
PaymentSucceeded | Payment state machine → Merchant webhook delivery, reconciliation projection | Outbox; merchant delivery signed, retried with backoff, event-id dedupe |
RefundSucceeded | Refund workflow → Merchant webhook, reports | At-least-once; immutable event id |
ReconciliationMismatchFound | Reconciler → Exception queue/alerting | Unique provider object + settlement window; never auto-adjust silently |
Failure modes
Provider succeeded, response lost
Failure
Confirm times out after provider charge.
Mitigation
Intent remains processing; query by stable provider reference/idempotency key and reconcile.
Duplicate merchant request
Failure
Retry could create a second payment/refund.
Mitigation
Merchant-scoped idempotency record binds key to request hash and stored result.
Duplicate/out-of-order webhook
Failure
Provider events repeat or arrive late.
Mitigation
Verify, store unique event id, and apply only legal monotonic state transitions; retrieve provider object if needed.
Refund race
Failure
Concurrent partial refunds exceed captured amount.
Mitigation
Lock payment/refundable row and include pending refunds in available amount.
Ledger worker crash
Failure
Payment changes without accounting entry.
Mitigation
State transition and balanced posting share one database transaction.
Platform/provider mismatch
Failure
Provider settlement differs from internal records.
Mitigation
Scheduled reconciliation creates explicit exception; humans resolve with audited reversal/adjustment.
Consistency and concurrency
Strong consistency / transactions
- Idempotency outcome
- legal payment transitions
- refundable amount
- balanced ledger posting
Eventual convergence
- Merchant webhook delivery
- provider convergence after ambiguity
- reports
- settlement reconciliation completion
Concurrency controls
- Unique merchant/idempotency key plus request hash
- Payment aggregate row lock for refund/capture limits
- Unique ledger source prevents double posting
- Webhook unique provider event id
- No distributed lock is a substitute for these constraints
Caching and background work
Safe to cache
- Public provider configuration by version
- Merchant display settings
Never a cache authority
- Payment state as command authority
- Available refundable amount
- Balances
- Idempotency result before durable completion
Payment reads may use a brief projection only if freshness is visible; all money decisions use PostgreSQL. Never cache secrets.
Background jobs
- Provider status reconciliation
- Webhook inbox processing
- Merchant webhook delivery with backoff
- Settlement-file reconciliation
- Idempotency retention cleanup
- Exception reporting
Security boundaries
Security claims here are design controls, not certification. Threat modeling and jurisdiction-specific review remain required.
- Use provider-hosted/tokenized collection; never store PAN or CVV
- Vault provider/API secrets and rotate with overlapping keys
- Merchant credentials are scoped, hashed where applicable, and tenant-bound
- Verify webhook signature over raw body plus timestamp; dedupe event id
- Object-level merchant authorization on every intent/refund/ledger read
- Audit refund, replay, exception, and credential actions
- Rate-limit by credential, merchant, IP, and risk signal; do not log payment tokens or PII
Observability
Logs
- Intent/attempt/refund transition with merchant, stable provider reference, idempotency key hash, and reason
- Webhook signature result, provider event id, inbox state, and legal-transition decision
- Ledger transaction source and account ids without tokens or customer payment data
Metrics
- Authorization success/decline/ambiguous outcome by provider reason class
- Payment-processing age and provider retrieval reconciliation latency
- Webhook receipt-to-apply age, duplicate rate, and merchant delivery backlog
- Ledger balance-check failures and settlement reconciliation exception count
Traces
- Trace confirm across idempotency reservation, provider call, state transition, ledger posting, and outbox
- Trace refund across refundable lock, provider ambiguity, webhook convergence, and reversal posting
Alerts
- Any unbalanced ledger transaction or posting invariant failure
- Processing payments exceed the ambiguity-resolution window
- Webhook or settlement reconciliation backlog threatens financial reporting
Testing strategy
| Layer | What it proves |
|---|---|
| Unit | Money/currency, state machine, idempotency hash, ledger balance |
| Integration | Real PostgreSQL locks/constraints, atomic posting, webhook inbox/outbox |
| Contract | Provider sandbox and signed merchant webhook versions |
| Concurrency | Same idempotency key, partial refund races, webhook/confirm race |
| Failure | Timeout after provider success, out-of-order events, worker crash |
| End-to-end/load | Intent → action → success → refund → reconciliation; workload stated explicitly |
Do not mock PostgreSQL constraints, transaction isolation, object-store semantics, or provider contract fixtures in the tests intended to prove those boundaries.
Deployment evolution
Stage 1
Private payment API, PostgreSQL, queue/workers, provider SDK, secret manager
Stage 2
Separated webhook ingress, command API and reconciliation workers; primary/standby database
Stage 3
Region/merchant cells with a single writer per payment; isolated ledger/reconciliation controls and replicated read reports
Scale evolution
These are architecture scenarios, not throughput claims. Each transition should follow measured workload and reliability pressure.
| Scenario | Architecture may evolve toward | Decision rule |
|---|---|---|
| 1K users | One API, PostgreSQL for intents, idempotency keys, and the ledger, a webhook worker, a reconciliation job | Write each state change and its balanced ledger entries in one transaction; every provider call carries a stable reference |
| 100K users | Scale stateless APIs/webhook ingest and workers independently; keep payment+ledger transaction on one primary | Scale webhook ingest separately so provider bursts cannot delay merchant API calls; the ledger stays on one primary |
| 1M+ users | Partition by merchant or payment home region; route all lifecycle commands to one owner and aggregate immutable events for reporting | Give each payment one home partition for its whole lifecycle; reporting reads an immutable event stream |