Components impacted

Management: Admin Console

Summary

On April 20, 2024, between 5:32 GMT and 6:57 GMT, Recent DNS Activity on the Admin Console became unavailable.

Shortly after the incident began, the Twingate on-call team received alerts regarding abnormal database activity. Workers on the clusters that manages DNS filtering logs starting seeing errors from the logs API, leading to excessive retries and database writes. To mitigate the issue, the DNS Log Streaming workers were temporarily disabled.

The root cause was identified as a malfunction in the DNS Filtering Log API caused by a problematic dependency upgrade. Consequently, viewing DNS filtering logs and analytics in the Admin Console was temporarily unavailable.

A rollback of the update was issued, and normal operations were restored at 6:57 GMT after which DNS filtering logs and analytics were available in the Admin Console.

Root cause

The DNS Filtering Log API went down due to a bad dependency upgrade.

Corrective actions

Already completed:

• Rectified Admin Console's infinite retry logic by enhancing the retrieval of DNS activity logs during error states.
• Optimized DNS Log Streaming retry and database write procedures to reduce unnecessary operations when no events are returned from the DNS Filtering API

Short-term:

• Improve the dependency upgrade process for the DNS FIltering API