Skip to main content

Known limits and future improvements

This page documents points that have not been load-tested yet, identified while reasoning through a burst scenario — for example, 1000 participants registering/activating a wallet at the same time at an event. None of this is a bug; these are sizing decisions that make sense today and deserve a second look before an event with a large crowd.

Per-IP rate limiting at event venues​

The authentication limiters on wallet-edge (REGISTRATION_RATE_LIMITER: 5/60s, LOGIN_RATE_LIMITER: 20/60s) key off a hash of the IP address when there's no token yet — see Authentication and profiles. That's reasonable for regular home internet, but this is an event platform: it's common for hundreds of participants to be behind the same venue Wi-Fi/NAT, sharing a single public IP. In that scenario, the per-IP limit can block legitimate participants, not just abuse.

Future idea: rethink the limiter key for this context — for example, combine IP with some device/session identifier, or make the limit configurable per event (bigger events, higher limit).

Regional database connection pool​

DATABASE_POOL_MAX on wallet-api is set to 5. Under a spike of thousands of simultaneous activations on a single instance, this turns into a queue waiting for a free connection — high latency before it turns into an actual error, but still a real bottleneck under an extreme burst. Worth a load test before assuming the current value handles the expected peak, and worth considering PgBouncer/a pooler if multiple horizontally-scaled instances (Cloud Run) add up to more connections than Supabase allows per project.

Outbox dispatcher: throughput under a burst​

This is the most important point to revisit. The dispatcher (OutboxDispatcherService, see the full mechanism in Overview → the global wallet directory) currently:

  • Runs every OUTBOX_POLL_INTERVAL_MS (5000ms).
  • Claims up to OUTBOX_BATCH_SIZE rows per cycle (10).
  • Delivers them sequentially, one at a time, within the batch:
for (const row of rows) {
  await this.deliver(row, lockToken, maxAttempts);
}

Back-of-envelope worst case: 10 deliveries per 5s cycle ⇒ at most ~2 events/second of sustained throughput per instance, not even counting the network time of each signed call to wallet-control-api (which lowers that number in practice, since delivery is sequential and each await blocks the next one). With 1000 wallets activated at once, ops.outbox_events would have 1000 pending rows, and draining all of them could take several minutes — the last person in line would only show up in GET /v1/wallet/wallets well after already having a fully active, usable wallet in the region.

Important: this does not block activation itself — that's synchronous and immediate in the regional database. The delay is only in the global discovery mirror, which is already documented as eventual. But several minutes of delay under a large burst is a lot more than the few seconds assumed today.

Future improvements, in order of effort​

  1. Deliver the batch in parallel (with a concurrency cap, e.g. Promise.allSettled in groups of N) instead of today's sequential for loop — an immediate throughput gain with no infrastructure change.
  2. Increase OUTBOX_BATCH_SIZE and/or dynamically shrink OUTBOX_POLL_INTERVAL_MS when the backlog is high (poll more aggressively only while there's a queue).
  3. Push-based notification (Postgres LISTEN/NOTIFY, or Supabase Realtime) to wake the dispatcher as soon as a row is inserted, instead of waiting for the next poll — this reduces the first event's latency, but doesn't replace the outbox's role as a delivery-guarantee safety net, as already discussed in this same architecture section.

None of these changes have been implemented — they're recorded here as next steps in case a load test confirms they're actually needed.