Summary

On Tuesday, March 19th, Between 4:58 PM UTC and 6:30 PM UTC an issue occurred with runs connecting to Snowflake warehouses using keypair authentication. Affected runs saw a dbt Core error message beginning with: “Could not deserialize key data”, which prevented runs from proceeding.

Impact

During the outage, runs attempting to connect to Snowflake using keypair authentication on dbt Core versions 1.5 and 1.6 failed to deserialize the private key used for authentication. All runs that could not connect to Snowflake errored.

We apologize for this outage and to every customer that was affected. We are making improvements to our testing process to ensure that a similar outage is unlikely in the future.

Root Cause

Code change

The root cause of the issue was a code change which altered the logic that dbt Cloud uses to generate dbt Core profiles (i.e. profiles.yml) for Snowflake connections using keypair authentication. We removed a private key path field that gets added in the generated dbt Core profile as we believed that it wasn’t required on dbt versions 1.5 and greater. We later identified a bug with core versions 1.5 and 1.6 that still required the field which was removed in the generated profile. A fix to not require this field in the generated profiles.yaml was added back in dbt version 1.7, which meant that runs passed in version 1.7 but were failing on dbt versions 1.5 and 1.6.

Mitigation

Our existing observability metrics helped us identify the time that the change made it to production and caused the runs to fail. We were able to look through code changes deployed at this time to identify which one in particular led to this issue. Once identified, the code change was rolled back and impacted jobs started executing successfully again.

Next Steps or Lessons Learned

Planned Remediation

Immediate mitigation and observability - ETA: End of Apr’24

• We are currently working on expanding our end-to-end test suite to exercise every combination of dbt Core version, adapter, and authentication method that’s supported in dbt Cloud.
• We will be adding additional log attributes to give us better observability into which adapters and dbt Core versions are impacted during incidents.