Skip to main content

Push notifications (proposal under discussion)

:::info Nothing on this page is implemented This is a design under discussion, recorded here to keep the conversation going — none of the endpoints, queues, or services described below exist in the code today. If (and when) this gets implemented, this page becomes the reference describing what actually exists, same as the other pages in this section. :::

Why this matters​

Not every charge/recharge happens with the participant looking at their own app. Two concrete cases where they're not in the app at the moment of the transaction:

  • Cashier-assisted top-up (Cashier → Top-up) — the cashier credits the wallet, the participant just hands over their phone/shows their face, they don't open the app.
  • Payment via printed QR + PIN (PDV → Payment via printed QR + PIN) — the participant doesn't even need the app installed for this flow; if they have it, it'd be good for them to know they were charged.

In dynamic payment via QR/NFC the participant is already inside the app approving, so they already know the outcome right away, from the approve call's own response — but we decided to notify anyway: it works as a persistent receipt, the same way a store still emails you a confirmation even after you've seen the total on the card machine. In other words: a top-up and any QR payment (printed+PIN or dynamic) trigger a notification, regardless of which channel ends up delivering it.

Decision 1: where the queue lives​

A similar mechanism already exists in the system: the outbox (ops.outbox_events + OutboxDispatcherService), which today only syncs wallet.activated to the global directory. Two options:

Option A — generalize the existing outbox (Postgres)​

Add more event types to the same table, swap the dispatcher for type-based routing. Simple to build, but inherits a limitation already documented: Known limits → outbox dispatcher throughput — sequential delivery, a fixed poll interval, everything running inside wallet-api's own process (a setInterval alive the whole time, using up the instance's resources even with nothing to deliver).

Option B — Cloudflare Queues (recommended)​

Moves the queue entirely outside wallet-api's process. Backoff retries, batching, and a Dead Letter Queue are all managed by Cloudflare — none of that needs to be written or maintained by us. Genuinely serverless: the consumer only runs when there's a message, no idle process.

Important constraint that shapes the design: messages can only be produced from inside a Worker (env.QUEUE.send(...), via a binding) — there's no HTTP API for an outside service (like wallet-api, which is NestJS on Cloud Run, not a Worker) to write directly to the queue. That's why wallet-edge sits in the middle: it's already a Worker, and the pattern of wallet-api calling it over a signed internal request already exists (see PaymentIntentEdgeNotifierService).

wallet-api (charge/top-up completes)
        │  signed internal call (same pattern as the payment-intent
        │  notifier, fire-and-forget, never blocks the HTTP response)
        ▼
   wallet-edge  ──produces──►  Cloudflare Queue
                                      │
                                      ▼
                       Worker consumer (push-based, batched)
                                      │
                     looks up the participant's token/contact
                                      ▼
                            wallet-control-api
                                      │
                         sends over the right channel
                                      ▼
                    Expo Push (today) / SMS / email (later)

Trade-off: one more network hop (wallet-api → wallet-edge → queue), and delivery is at-least-once — the consumer needs to be idempotent (same principle already used for every Idempotency-Key in the system), so a retry doesn't send the same push twice.

Scale: 5,000 messages/second per queue — huge headroom over the volume we're planning for (5,000 transactions/minute).

Decision 2: does the consumer Worker do everything, or just relay?​

Once the message reaches the consumer (Worker), who decides the channel and actually sends the notification?

Option A — Worker does everything​

The consumer itself looks up the token in wallet-control-api and calls Expo's API directly from the edge.

  • ❌ Puts business logic (which channel, which sender, when to notify) inside wallet-edge — which today is deliberately "dumb": it only routes, does preliminary auth, and signs, never holds data or decides business rules (see Architecture overview). Mixing notification logic in there breaks that separation.
  • ❌ That logic would live in Workers/TypeScript infra code, outside the NestJS conventions used for everything else (Zod, repository/service, the same tests as always).

Option B — Worker as a relay to a webhook (recommended)​

The consumer only handles queue mechanics (batching, ack, retry) — for each message, it calls a signed internal endpoint on wallet-control-api, and decides ack/retry based on the HTTP response.

  • ✅ All notification logic (looking up the token, picking the channel, calling the right sender) lives in wallet-control-api — same language, same tests, same way we build everything else.
  • ✅ wallet-edge stays thin, doing only what it already does today.
  • ✅ Still gets everything Cloudflare Queues offers (backoff retries, DLQ, batching, no idle process) — the Worker is still the one actually processing the batch; the webhook is just what it calls internally.
  • ❌ If wallet-control-api is down/slow at that moment, the call fails and Cloudflare retries later — that's exactly the managed retry we want, not really a downside.

Decision 3: separate "what happened" from "how to notify"​

The event recorded on the queue is generic (wallet.charged, wallet.recharged) — the consumer decides the channel through a pluggable per-channel interface (ExpoPushSender today; SmsSender/EmailSender later, same interface). That way, adding a new channel later touches nothing in wallet-api — it only ever emits the event, it never knew and never needs to know who notifies or how.

Full example flow: a top-up​

To make it concrete, the full path of a cashier top-up — in two clearly separated phases: the top-up itself (synchronous, already exists today, nothing changes) and the notification (asynchronous, new, never blocks the first).

Phase 1 — the top-up itself (synchronous, unchanged)​

Cashier app
     │  POST /v1/events/{eventId}/cashier/topups
     ▼
wallet-edge  ── single public entry point, routes by eventId → right region
     │  signed proxy (X-Origin-Signature)
     ▼
wallet-api (regional)
     │  CashierTopupsController → CashierTopupsService.recharge()
     │  transaction: lock wallet FOR UPDATE → ledger.transactions
     │  (type='recharge') + 2 ledger.entries → wallet_balances →
     │  idempotency key → COMMIT
     ▼
HTTP response comes back (walletId, availableAmountMinor, replayed)
     │
     ▼
wallet-edge → Cashier app

Phase 2 — the notification (new, asynchronous)​

Right after the COMMIT (the same spot where completeIdempotencyKey is called today, before the return) — awaited with a short timeout, never throws, never delays the response to the cashier (same pattern as PaymentIntentEdgeNotifierService):

wallet-api (right after the top-up commits)
     │  signed internal call (same scheme already used today,
     │  fire-and-forget, short timeout)
     ▼
wallet-edge — new route: POST /internal/v1/notifications/enqueue
     │  verifies the signature (same pattern as payment-intent-notify.ts:
     │  verifyControlPlaneNotifyRequest)
     │  env.NOTIFICATIONS_QUEUE.send({ type: "wallet.recharged", userId, eventId, amountMinor, ... })
     │  responds 204 IMMEDIATELY — doesn't wait for the queue to be consumed
     ▼
204 response comes back to wallet-api (just confirms it was ENQUEUED)

The top-up's HTTP response to the cashier was already delivered before or alongside this — the cashier never waits on Phase 2 for anything.

From here on, disconnected from any HTTP request — Cloudflare itself decides when to process, in batches:

Cloudflare Queue
     │  Cloudflare invokes wallet-edge's queue handler
     │  (same Worker, `queue()` export — an independent invocation,
     │  not called by any of our own code)
     ▼
wallet-edge (consumer) — Decision 2, Option B
     │  for each message in the batch:
     │  signed internal call → wallet-control-api
     ▼
wallet-control-api — new route: POST /internal/v1/notifications/dispatch
     │  looks up the userId's push token(s) (new table, app.push_tokens)
     │  decides the channel (Expo Push today) → calls Expo's API
     │  responds 200/204
     ▼
wallet-edge: msg.ack() on success, msg.retry({delaySeconds}) on failure
     (Cloudflare handles backoff/DLQ automatically)
     ▼
Expo → APNs/FCM → notification shows up on the participant's phone

A QR payment (printed+PIN or dynamic) follows the same design, just swapping Phase 2's trigger point: after PdvPaymentsRepository.chargeWallet (printed) or WalletPaymentIntentsRepository.approve (dynamic), instead of after recharge.

Key points:

  • The financial transaction never waits on the notification — two completely separate HTTP lifecycles.
  • If wallet-edge is down at the moment of enqueueing, the notification simply doesn't happen for that transaction — same graceful degradation already accepted today for the payment-intent notifier; the transaction itself is never affected.
  • If wallet-control-api is down/slow at the moment of consumption, Cloudflare retries on its own, with none of our own code running in a loop.

What triggers a notification (and what doesn't)​

  • Triggers: a completed top-up, an approved payment — printed QR or dynamic, both — the person gets a confirmation/receipt.
  • Probably doesn't trigger (still open): declines (insufficient balance, locked PIN, etc.) — whoever is being declined is already there, face to face with the cashier/PDV, and already knows the outcome on the spot; an extra push adds nothing.

Where the device token lives​

A push token is global per user (not per event/region) — it makes sense for it to live in wallet-control-api, which already owns the participant's identity, not in the regional wallet-api. Needs a new table (something like app.push_tokens — a participant can have more than one device) and an endpoint for the app to register the token after login.

Open questions​

  • Exact event catalog — just top-up/payment, or also a refund once that exists?
  • Explicit consent (OS-level permission) is already standard Expo/RN behavior — but does it need a user preference toggle ("I want to receive charge notifications") from day one, or can that come later?
  • Notification content: showing the charged amount is fine, but showing the current balance on the lock screen is a privacy choice still to be made.
  • SMS/email depend on a verified phone/email — today the participant only registers a phone (OTP); email doesn't even exist as a field yet.