Database Connection Issues
Incident Report for Twingate
Postmortem

Components impacted

Control Plane: Authn, Authz, Connector Heartbeat

Management: Admin Console, Public API, Identity Providers Sync

Summary

On Dec 11, 2023 between the hours of 5:02 pm UTC and 6:24 pm UTC, Twingate received network connectivity alerts at 3 distinct times, each for a few minutes. After investigation, it was identified that a change in our application had caused our connections to exceed the allowed maximum.

We reverted the changes at 6:24 pm, after which the increased connections receded and our connection counts returned to healthy counts.

Root cause

The Twingate database connections exceeded the maximum allowed.

Corrective actions

Already Completed:

  • Adjusted capacity to keep the number of connections at a healthy number

Short Term:

  • Increase the maximum number of connections allowed
  • Add alerting for connection utilization
Posted Dec 12, 2023 - 05:44 UTC

Resolved
Summary

On Dec 11, 2023 between the hours of 5:02 pm UTC and 6:24 pm UTC, Twingate received network connectivity alerts at 3 distinct times, each for a few minutes. After investigation, it was identified that a change in our application had caused our connections to exceed the allowed maximum.

We reverted the changes at 6:24 pm, after which the increased connections receded and our connection counts returned to healthy counts.

(See Postmortem for more details)
Posted Dec 11, 2023 - 17:00 UTC