Summary

On June 3rd at 3 AM UTC, Twingate started a regular Kubernetes upgrade on its main cluster. This maintenance is usually done once a month and has been performed successfully many times prior to this upgrade. It includes a version upgrade of the cluster followed by a version upgrade of node-pools, completed one at a time.

Around 4:01 AM UTC, HTTP 502 errors started on our cloud load balancer instance indicating an issue with the service. While these errors were a small portion of the overall volume of requests at first, around 4:15 AM the system was fully overloaded and it turned into a full outage. Shortly after, we failed to our standby cluster in a different region of our cloud provider, but saw the same issue happening on our standby cluster too. We downgraded our active clusters node pools to the previous Kubernetes version. This added extra capacity and then we failed back to our active cluster at 5:05 AM UTC. Recovery started immediately and the service was fully recovered at 5:10 AM UTC.

Root cause

After a detailed investigation, we found that during the upgrade, network connectivity between internal components was not stable, triggering failures and retries. As a result of our application being overloaded, we failed to answer load balancer health checks which caused the 502 errors. We are working with our cloud provider to analyze why the network instability happened during the upgrade.

Corrective actions

We have initiated a number of improvements:

• Completed: We increased our main application capacity, tuned application and network settings between mentioned services, upgraded our in-memory key-value store, and added PDB (pod destruction budget).
• Short-term: We will continue to tune the application and network settings between various components of Twingate. We found a bug with how our our client handles 502 errors and we are working on our client to handle the 502 errors better.
• Medium-term: We are looking into two major changes: 1) implementing Circuit Breaker functionality so our main application can stay up when a downstream service goes down, and 2) implement a multi-region active-active setup on our cloud provider, which will enable us to better control Kubernetes upgrades (as well as other code and configuration changes).