We have implemented a stable workaround for an issue that was impacting jobs in projects connected to Snowflake where the job was running multiple dbt threads and the project was referencing multiple databases in Snowflake in sources and/or models.
This issue intermittently caused jobs to fail in Snowflake-connected projects between approximately 0450 UTC on Wednesday June 28 and 2300 UTC on Friday June 30.
We are continuing to work with Snowflake to diagnose the root cause of this incident and determine an appropriate long term fix.
If you continue to experience job runs that fail with limited or missing logs in dbt Cloud, please contact dbt Labs Support via the support chat in dbt Cloud or by emailing support@getdbt.com. Please include the link to the failed run in dbt Cloud in your message.
Posted Jul 07, 2023 - 16:49 EDT
Monitoring
After investigation we have identified a workaround that is preventing jobs from failing while we continue our investigation into the root cause and a permanent mitigation.
If you continue to experience jobs with a thread count of greater than 1 failing shortly after the invocation of a dbt command with truncated debug logs, please contact support via the in-product Chat or by emailing support@getdbt.com with the URL of the affected job for further investigation.
Posted Jun 30, 2023 - 19:58 EDT
Update
We continue to investigate the root cause of failures of less than 0.5% of runs with interrupted logs. These failures typically happen within the first minute of the invocation of a dbt command and debug logs will show multiple threads stalling just after attempting to run the show terse schemas SQL statement.
If you are experiencing consistent failures of the same job and are unable to successfully manually re-run the job, please contact support via the in-product Chat option or via email at support@getdbt.com and supply the url to the impacted job for further investigation.
Posted Jun 29, 2023 - 22:44 EDT
Update
We continue to investigate the root cause of failures of less than 0.5% of runs with interrupted logs. These failures typically happen within the first minute of the invocation of a dbt command and debug logs will show multiple threads stalling just after attempting to run the show terse schemas SQL statement.
If you are experiencing consistent failures of the same job and are unable to successfully manually re-run the job, please contact support via the in-product Chat option or via email at support@getdbt.com and supply the url to the impacted job for further investigation.
Posted Jun 29, 2023 - 22:40 EDT
Update
A small percentage of runs are intermittently failing because of underlying infrastructure issues starting Wed 28th 4:30AM UTC. We noticed significant increase in segmentation faults across our systems. Re-running the job should resolve the issue. We are working with our infrastructure providers to resolve the underlying os level issues.
Posted Jun 28, 2023 - 22:29 EDT
Update
We continue our investigation into identifying the issue and working towards a resolution. We will be updating shortly.
Posted Jun 28, 2023 - 22:04 EDT
Investigating
We have received reports of users unable to view logging after a job run fails. We're currently investigating and will be updating shortly.
Posted Jun 28, 2023 - 18:17 EDT
This incident affected: North America (N. Virginia) (Scheduled Jobs), Europe (Frankfurt) (Scheduled Jobs), and Australia (Sydney) (Scheduled Jobs).