Summary

On November 22, 2024, from 4:13 AM to 6:48 AM UTC, job runs relying on PrivateLink connections failed across multiple tenants. This was caused by a code change that prevented the job execution service from properly using PrivateLink for warehouse connections. The issue was detected via a monitoring alert, and a rollback resolved the problem.

Impact

• During the impact window (4:13 AM to 6:48 AM UTC), all jobs relying on PrivateLink connections failed to execute for dbt Cloud accounts in AWS multi-tenant and multi-cell environments.
• While this accounted for approximately 1% of all job runs across our entire customer base, we recognize that customers that heavily dependent on PrivateLink may have been significantly impacted.
• Impacted job runs failed with errors connecting to data warehouses and could be successfully re-run after the rollback.

Root Cause

A code change to environment variables caused incorrect PrivateLink configuration to be passed to the job execution service. This resulted in jobs runs not being able to connect to PrivateLink endpoints and ultimately failing with data warehouse connection errors.

Detection and Mitigation:

The issue was detected by our monitoring system. Once the faulty code change was identified, the issue was resolved via a rollback.

Total resolution time was longer than usual due to the incident occurring during the holiday deployment freeze, which required additional processes to lift the freeze and initiate the rollback.

Prevention:

• Completed

  • The faulty code change was reverted following the release rollback.
  • Validation mechanisms were implemented to prevent similar misconfigurations in the future.

• Longer-term (by end of Jan 2025):

  • Implement additional tests for PrivateLink scenarios to catch similar issues before they reach production.
  • We are in the process of larger architectural improvements, which include decoupling job scheduling functionality from other dbt Cloud services. This change will result in cleaner configurations management and will help prevent this kind of issues in the future.

We sincerely apologise for this outage and to every customer that was impacted. We understand that you rely on the dbt Cloud application, especially job execution, as a key tool. We are confident that these measures will effectively prevent similar incidents in the future and ensure a more stable experience for all our users.