This incident has been resolved. All production runs are now running as per normal.
Note that past stalled/cancelled runs would need to be manually retried. Please contact Support at support@getdbt.com or via the dbt Cloud UI if you continue to experience issue with your runs being stuck at the 'Starting' state and/or timing out after 30 min of inactivity.
Posted Jul 08, 2024 - 21:46 EDT
Monitoring
We have implemented a fix and production runs are working as expected. We will monitor the situation for the next 30 minutes to ensure everything is back to normal.
Note that any affected runs that were idle for more than 30 minutes, would be automatically cancelled and you will see the error message: "This run timed out after 30 minutes of inactivity. Please retry the run, or contact support if this job continues to fail." They will not be retried automatically.
Posted Jul 08, 2024 - 21:11 EDT
Update
We are continuing to investigate this issue.
Posted Jul 08, 2024 - 20:27 EDT
Investigating
We're investigating an issue with our job scheduler that is causing stale run pods to get canceled automatically. This is impacting production runs in EMEA MT. The team is working on a resolution and we will provide updates at approximately 30 minute intervals or as soon as new information becomes available.
Posted Jul 08, 2024 - 20:25 EDT
This incident affected: Europe (Frankfurt) (Scheduled Jobs).