Webhook Notification Delivery Delays and Missing Notifications
Incident
Report for Currencycloud
Resolved
The issue affecting webhook notification delivery has now been resolved. All queued webhook notifications, including the backlog accumulated during the incident, have been successfully processed and delivered.
Throughout the recovery period, Engineering investigated intermittent processing behaviour that caused backlog processing to pause and subsequently resume. Recovery actions were completed successfully, processing stability was restored, and queue volumes returned to normal operating levels.
Underlying transactions continued to process normally throughout the incident and no data loss has been identified. Recovery validation has been completed and the service is operating as expected.
Throughout the recovery period, Engineering investigated intermittent processing behaviour that caused backlog processing to pause and subsequently resume. Recovery actions were completed successfully, processing stability was restored, and queue volumes returned to normal operating levels.
Underlying transactions continued to process normally throughout the incident and no data loss has been identified. Recovery validation has been completed and the service is operating as expected.
Update
Webhook processing remains active and delayed notifications continue to be delivered. Overall queue volumes have reduced by approximately 85%, including both the original backlog and newly generated webhook events.
Engineering continues to investigate the intermittent behaviour causing backlog processing to pause and subsequently resume. Whilst recovery is progressing and queue volumes continue to reduce, processing rates have varied during the recovery period and investigation remains ongoing to identify the underlying cause and restore consistently stable processing.
Underlying transactions continue to process normally and no data loss has currently been identified.
Engineering continues to investigate the intermittent behaviour causing backlog processing to pause and subsequently resume. Whilst recovery is progressing and queue volumes continue to reduce, processing rates have varied during the recovery period and investigation remains ongoing to identify the underlying cause and restore consistently stable processing.
Underlying transactions continue to process normally and no data loss has currently been identified.
Update
Webhook processing remains active and delayed notifications continue to be delivered. Overall queue volumes have reduced by approximately 70%, including both the original backlog and newly generated webhook events.
Engineering continues to investigate the intermittent behaviour causing backlog processing to pause and subsequently resume. Whilst recovery is progressing and queue volumes continue to reduce, processing rates have varied during the recovery period and investigation remains ongoing to identify the underlying cause and restore consistently stable processing.
Underlying transactions continue to process normally and no data loss has currently been identified.
Engineering continues to investigate the intermittent behaviour causing backlog processing to pause and subsequently resume. Whilst recovery is progressing and queue volumes continue to reduce, processing rates have varied during the recovery period and investigation remains ongoing to identify the underlying cause and restore consistently stable processing.
Underlying transactions continue to process normally and no data loss has currently been identified.
Monitoring
Webhook processing has resumed and delayed notifications are being delivered.
Engineering is continuing to investigate an intermittent issue that is causing backlog processing to pause and restart, with recovery efforts focused on maintaining stable processing until all queued notifications have been delivered.
Engineering is continuing to investigate an intermittent issue that is causing backlog processing to pause and restart, with recovery efforts focused on maintaining stable processing until all queued notifications have been delivered.
Update
The production fix remains deployed and has addressed the original issue that caused webhook notifications to accumulate. However, processing of the retained backlog has stopped again, and delayed notifications are not currently progressing through the queue.
Engineering Teams are actively investigating a secondary issue affecting backlog processing and is working to restore delivery while preserving all queued webhook events.
Underlying transactions continue to process normally, but clients will continue to experience delayed webhook notifications until backlog processing resumes. No data loss has currently been identified.
We will provide a further update once sustained processing and queue reduction have been restored.
Engineering Teams are actively investigating a secondary issue affecting backlog processing and is working to restore delivery while preserving all queued webhook events.
Underlying transactions continue to process normally, but clients will continue to experience delayed webhook notifications until backlog processing resumes. No data loss has currently been identified.
We will provide a further update once sustained processing and queue reduction have been restored.
Identified
The production fix remains deployed and addressed the original issue that caused webhook notifications to accumulate. However, processing of the existing backlog stopped at approximately 14:40 BST, and delayed notifications are not currently progressing through the queue. Engineering is actively investigating the cause and assessing the safest recovery action while preserving all queued webhook events.
Current evidence suggests this remains related to processing the unusually high backlog volume rather than a new issue affecting underlying transactions. Transactions continue to process normally, but clients will continue to experience delayed webhook notifications until backlog processing resumes. We will provide a further update once sustained processing and queue reduction have been restored.
Current evidence suggests this remains related to processing the unusually high backlog volume rather than a new issue affecting underlying transactions. Transactions continue to process normally, but clients will continue to experience delayed webhook notifications until backlog processing resumes. We will provide a further update once sustained processing and queue reduction have been restored.
Update
The production fix remains in place and has addressed the original issue that caused webhook notifications to accumulate. During recovery, processing of the existing backlog has slowed due to database load, and Engineering is investigating the safest way to resume and accelerate clearance without placing additional pressure on the service.
This is not currently considered a recurrence of the original incident. It relates to processing the unusually high volume of notifications already queued. Underlying transactions continue to process normally, but clients may continue to experience delays in receiving webhook notifications while backlog recovery is completed.
This is not currently considered a recurrence of the original incident. It relates to processing the unusually high volume of notifications already queued. Underlying transactions continue to process normally, but clients may continue to experience delays in receiving webhook notifications while backlog recovery is completed.
Monitoring
We have mitigated the issue affecting webhook notification delivery. A targeted fix has been successfully deployed to production, and webhook notifications are now being processed and delivered. The queue of delayed notifications is reducing, and current recovery indicators show the service is operating as expected.
Underlying transactions, funding events and bank transfers continued to process successfully throughout the incident. Current evidence indicates that webhook notifications were delayed and retained for delivery rather than lost, although delivery validation remains ongoing. No client action is required at this time.
Underlying transactions, funding events and bank transfers continued to process successfully throughout the incident. Current evidence indicates that webhook notifications were delayed and retained for delivery rather than lost, although delivery validation remains ongoing. No client action is required at this time.
Investigating
We are currently investigating an issue affecting webhook notification delivery for some Currencycloud customers.
Customers may experience delayed or missing webhook notifications for transactions, funding events, and bank transfers. We have received reports from multiple customers and are actively working to determine the scope and cause of the issue.
Our teams are investigating with the highest priority and will provide updates as they become available.
We apologize for any inconvenience and appreciate your patience while we work to resolve this issue.
Customers may experience delayed or missing webhook notifications for transactions, funding events, and bank transfers. We have received reports from multiple customers and are actively working to determine the scope and cause of the issue.
Our teams are investigating with the highest priority and will provide updates as they become available.
We apologize for any inconvenience and appreciate your patience while we work to resolve this issue.
This incident affected: API.