Elevated API Errors
Postmortem
Overview
An elevated error rate was observed across key API services, temporarily resulting in degraded performance. The issue was promptly resolved following a configuration rollback.
Client Impact
A number of Clients experienced intermittent issues creating new beneficiaries and encountered delays when interacting with payment-related endpoints. The disruption was temporary and confined to a limited period, during which some services remained partially available while others were intermittently affected.
Root Cause
A Redis authentication change to IAM credentials prevented successful connections, causing downstream failures. The resulting database connection exhaustion impacted multiple dependent services.
Remediation
The Redis configuration was reverted to restore connectivity. Additional health checks and investigation into IAM failures are planned to prevent recurrence.