We've implemented a root cause fix for this issue, and we've completely flushed the queue. Individual project queues may still be catching up, but scheduled runs are kicking off normally.
Our engineering team is continuing to monitor this incident.
Posted Mar 17, 2021 - 15:24 EDT
Update
We are continuing to work on this issue. No new runs in the last 30 minutes have failed to kick off, but a number of runs are still stuck in a bad state. Our engineers are working to flush the queue now. We'll post another update shortly.
Posted Mar 17, 2021 - 14:56 EDT
Identified
We've identified the source of the issue and are working to mitigate and deploy a root cause fix.
Posted Mar 17, 2021 - 14:09 EDT
Investigating
We've investigating an infrastructure issue that is causing a small percentage of scheduled runs to exit with a "timed out after 30 minutes of inactivity" message.
Posted Mar 17, 2021 - 13:51 EDT
This incident affected: North America (N. Virginia) (Scheduled Jobs).