Missing Webhooks
Incident
Report for Currencycloud
Postmortem
Incident Summary
On 17 June during a routine system upgrade, we encountered unexpected issues that disrupted how messages were processed and delivered across some of our services.
Root Cause
- A Debian OS upgrade introduced compatibility issues affecting message handling
- Our notifications system slowed down due to an inefficient database query
- Some services had trouble maintaining stable connections to our messaging system (RabbitMQ)
- Automatic retries overwhelmed the system, eventually redirecting messages into a backup queue
Remediation
- Identified and fixed the database query causing the bottleneck
- Adjusted messaging configurations to stabilize service connections
- Increased system resources and improved retry logic to reduce congestion
- Cleared affected queues and ensured all valid messages were processed
Service restored same day, with impacted messages reissued between 18th and 20th June.
Thank you for your patience as we worked to address the issue. If you're still experiencing any issues with missing messages, please don't hesitate to contact our support team directly.
Resolved
The incident has been resolved. All missing webhooks have been resent successfully.
Update
We are continuing to work on missing webhooks for clients
All Funding webhooks have been resent and will not be sent again.
All Funding webhooks have been resent and will not be sent again.
Monitoring
Webhooks have been resent, incident will remain open until all checks have been completed.
Identified
We are in the process of resending the missing notifications; however, this may result in some clients receiving duplicate webhooks and emails.
We apologies for any inconvenience this may have caused.
We apologies for any inconvenience this may have caused.
Update
We are working to identify missing webhooks so they can be resent, this work is ongoing
Investigating
Due to the incident on the 17th June, we are investigating some clients missing important webhook notifications, after incident was mitigated.
The team is working to identify the missing webhooks so they can be resent
The team is working to identify the missing webhooks so they can be resent
This incident affected: Notifications.
