Allocator API is down

Root Cause Analysis — July 13, 2026
Incident summary
On July 13, 2026, we experienced two service interruptions related to the same underlying issue:
07:37–08:36 UTC
13:52–14:12 UTC

Root cause:
The incident began when our Kubernetes cluster stopped forwarding logs due to an issue with Fluentd, our log collection service.
To restore log forwarding, we initiated a non-disruptive upgrade of the Kubernetes node pools.
Based on the expected behavior of this procedure, the upgrade was not anticipated to affect service availability.
When the upgrade reached the node pool hosting ScyllaDB, Fluentd began processing a backlog of accumulated logs.
This caused an unexpected increase in CPU and memory consumption on the affected nodes.
As a result, the cluster temporarily lacked sufficient resources to schedule the ScyllaDB pods, leading to service unavailability.
Once the node-pool upgrade had started, the operation needed to complete before we could safely apply further changes, this extended the recovery time by approximately 20 minutes.
During recovery, we developed a remediation plan to increase the resources available to the ScyllaDB workloads.
While deploying that change, our Terraform controller reconciled the environment to its previous configuration and
reverted part of the Fluentd update before the remediation was fully applied. This caused the second service interruption.

Resolution and preventive measures:
We have implemented a permanent remediation that:
- Provides additional CPU and memory headroom for all database workloads.
- Implementing BLUE/GREEN deployment strategy for database workloads.
- Maintains a dedicated fallback node pool so database pods can continue to be scheduled when their primary node pool is unavailable or resource-constrained.
- Prevents infrastructure reconciliation from reverting operational changes while a related remediation is being deployed.
- Reduces the likelihood that log-backlog processing can compete with critical database workloads for resources.
We regret the disruption caused by these incidents. We are continuing to review our upgrade, capacity-management, and infrastructure-reconciliation procedures to reduce both the likelihood and impact of similar events.