Summary

On July 21st, multiple customers in the AU MT region experienced job runs not starting and subsequently timing out after 30 minutes. This issue was traced back to a RabbitMQ exchange in the AU region not routing messages correctly between our job scheduling services.

Since this incident, we have implemented several enhancements to our systems to prevent such outages, including additional alerting and monitoring to enable faster detection and resolution of similar issues in the future.

Impact

• 47 customer accounts across AU were impacted during this outage
• In total, 818 out of 5090 job runs (approximately 16%) timed out across the impacted customers.
• Impacted runs can be manually restarted. Scheduled runs will continue to trigger normally.

Root Cause

We currently use RabbitMQ as a messaging broker for job orchestration. During the outage, two of the RabbitMQ exchanges in the AU region failed to route messages properly between our job scheduling services due to an unexpected behavior in a custom routing plugin. This plugin, designed to optimize message distribution, encountered an edge case that affected its core functionality. Consequently, a portion of the job runs were never queued for execution and they were subsequently canceled by a cleanup service after 30 minutes.

Unfortunately, our system monitors did not detect the condition of jobs not being properly queued and thus the issue was only identified and resolved after receiving reports from customers.

Mitigation and Prevention

• Immediate Steps: We addressed the issue by restarting the backend queueing and job scheduling services, which successfully restored normal operations.
• Long-term Solution: We are initiating a change to our backend scheduling architecture so that it is no longer reliant on a custom routing plugin. This change will align with the architecture already implemented in other areas of our platform, which has demonstrated superior stability.
• Alerting and Monitoring: We have implemented a mechanism for alerting and monitoring to immediately detect similar conditions in the future. This approach will significantly improve our ability to identify and mitigate any potential issues before they impact services.

We sincerely apologize for this outage and to every customer that was impacted. We understand that you rely on the dbt Cloud application, especially job execution, as a key tool. We are confident that these measures will effectively prevent similar incidents in the future and ensure a more stable experience for all our users.