Overview
An availability issue affected critical API endpoints, causing temporary service degradation. The issue was promptly resolved following a rollback of a recent internal change.
Client Impact
Between 10:14 AM and 10:51AM GMT on 5th November, a number of API endpoints experienced intermittent failures. As a result, small number of clients encountered issues when attempting to connect via API, with requests failing to process successfully within the expected time window. This led to retries and delays in workflow execution. While not all endpoints were impacted for the full duration, access service was reduced during this period.
Root Cause
The incident was caused by an unplanned production rollout of a change that disrupted the service. Retry mechanisms overwhelmed the system, leading to saturation and intermittent failures.
Remediation
The change was rolled back immediately to restore service. Monitoring was initiated to ensure continued system stability.
Currencycloud Confidential