On July 21st, multiple customers in the AU MT region experienced job runs not starting and subsequently timing out after 30 minutes. This issue was traced back to a RabbitMQ exchange in the AU region not routing messages correctly between our job scheduling services.
Since this incident, we have implemented several enhancements to our systems to prevent such outages, including additional alerting and monitoring to enable faster detection and resolution of similar issues in the future.
We currently use RabbitMQ as a messaging broker for job orchestration. During the outage, two of the RabbitMQ exchanges in the AU region failed to route messages properly between our job scheduling services due to an unexpected behavior in a custom routing plugin. This plugin, designed to optimize message distribution, encountered an edge case that affected its core functionality. Consequently, a portion of the job runs were never queued for execution and they were subsequently canceled by a cleanup service after 30 minutes.
Unfortunately, our system monitors did not detect the condition of jobs not being properly queued and thus the issue was only identified and resolved after receiving reports from customers.
We sincerely apologize for this outage and to every customer that was impacted. We understand that you rely on the dbt Cloud application, especially job execution, as a key tool. We are confident that these measures will effectively prevent similar incidents in the future and ensure a more stable experience for all our users.