8:45am UTC: All queues have been processed, we are back to normal. Our engineering team is continuing to monitor the system, and to implement changes which can help us avoid these types of issues in the future.
8:00am UTC: We have finished processing most of the backlogs. Automations and welcome campaigns should not experience delays. Reporting of sends, opens and clicks are up-to-date (delivered and bounced metrics are delayed by up to 1 hour)
6:20am UTC: It appears that AWS has resolved the underlying issue. We are still processing the overnight backlog of data. Email and SMS sends should be working, however reporting is still delayed for both the overnight and the new sends. Welcome campaigns may also be delayed for up to 1 hour.
5:30am UTC: AWS has started resolving the underlying issue, we are starting to process the overnight backlog of data.
9:46pm UTC: Reporting is still delayed. We are waiting for AWS to resolve the underlying Kinesis issue.
8:20pm UTC: All queued emails have been processed, emails should not be delayed. However, reporting is still delayed.
7:50pm UTC: We've increased the pace of email sends, there should not be more than 10 minute delay in emails
7:20pm UTC: We've resolved issues with increased latencies on our webhooks and frontend servers. Reporting is still delayed, it is highly connected to the failing AWS components.
6:00pm UTC: Reporting is still delayed, and the reports pages are showing zeros even though emails are in fact being sent. This issue will probably only be resolved once AWS is fully back online.
5:20pm UTC: Email sends are working, however reporting is still delayed so it may take a long while to see the numbers (including sends) in the reports.
4:00pm UTC: We have worked around the issue of redirecting email and SMS links, and they should be working properly now.
~Delays or failures in redirecting email and SMS links~ (Fixed)
Delays sending out emails and SMS's (Fixed)
Delays in triggering automations, including welcome campaigns
Delays in reporting: Reporting for emails and SMS campaigns may be delayed for up to 1-2 hours
For older plugins, could cause a slowdown on site responsiveness. If you see this being the case, please turn off sever-side webhooks.