Stopping the Flood: Avoiding Queue Exhaustion

Queueing Theory

If each worker processes jobs at an average rate of and there are workers, the combined processing rate is . For the queue to remain stable over time, that rate must exceed the average arrival rate (). Utilization, , must stay below 100%.

Averages hide variation. A false positive may close quickly, while an endpoint compromise takes much longer. A flood can push arrivals above capacity even when average utilization is low. For a single-server queue approaching full utilization from below, Kingman's formula approximates average waiting time [1]:

Here, is average service time, and and describe variation in interarrival and service times. This single-server approximation does not estimate delays for a multiworker SOC or a persistently overloaded queue. It shows how waiting time rises sharply near full utilization, especially when work is unpredictable.

Queueing delay depends on four variables: arrival rate, processing rate, arrival variability, and processing variability.

Spare capacity costs money, whether it is people, agents, CPU, or RAM. Security Operations is highly cost-sensitive, so excess staffing or compute is a tough sell.

Every dot in the interactive is one job; a rose outline marks critical work. Scrolling changes policies without clearing the queue. Sandbox lets you adjust the parameters yourself.

Why Resiliency Matters

D&R cannot schedule arrivals like a factory ordering parts. Misconfiguration and malicious activity can flood the system, and a malware campaign may also disrupt the infrastructure needed for response. If SIEM and SOAR are critical to remediation, their availability needs protection.

A queue buys time by separating arrival from processing. It does not create capacity. If 8,000 jobs arrive per second and the system processes 750, a one-minute flood leaves 435,000 jobs waiting, assuming there is room to retain them. If arrivals then fall to 500 per second, only 250 jobs per second of capacity remain to drain that backlog. Recovery takes 29 minutes after the flood has ended. A queue that cannot hold the work loses it sooner.

An alert flood can exhaust memory, storage, or connections until the system stops working. If the SOAR goes down, responders lose automation just as the workload increases. The outage adds manual work and delays containment.

Healthy infrastructure can still fall behind. The four approaches in my talk were to do less work, process it faster, preserve critical capacity, and bound the damage when something fails.

Do Less Work (Reduce λ)

Compaction reduces the number of jobs, admission control limits what enters processing, and deferral moves work to a later time. Shedding discards work that cannot be retained or processed; I'll cover that under failure order.

Compaction

Compaction replaces several units of work with fewer units that preserve the information needed for an investigation. A hundred related alerts might become one case with a hundred pieces of evidence. The original evidence still needs to be available. The queue illustration treats each compacted case as one job; in production, a larger case may also take longer to process.

Risk-Based Alerting

Risk-based alerting can reduce the number of low-fidelity findings that become cases. Instead of sending every weak signal to an analyst or playbook, we can accumulate risk against an entity and alert when the combination warrants investigation. This shifts some processing upstream and can make the resulting cases more useful.

For teams still developing their detection coverage, RBA can provide some of the benefits of composite detections without maintaining a separate rule for every combination. It still needs maintenance: risk scores depend on sensible windows, thresholds, and treatment of severe individual events. An isolated high-impact event should not disappear because it failed to accumulate enough points.

Deduplication

Deduplication removes repeated instances of the same finding. A retried webhook or several connectors reporting the same alert should not create several investigations. This is often a relatively cheap way to reduce arrival rate, but the identity and time window matter. Matching only on an indicator can merge unrelated incidents, while matching every field may miss duplicates because timestamps or delivery metadata differ. Keep the original evidence, define what counts as the same event, and make the operation safe to repeat.

Coalescence

Coalescence groups related findings rather than exact duplicates. Several events on the same endpoint might belong in one investigation even when each event is distinct. This saves repeated enrichment and lets the investigator see the sequence together. It also changes the unit of work: one case may become substantially larger. A useful grouping policy therefore needs a boundary, such as an entity, a time window, or an incident, and a way to reopen or split a case when the evidence warrants it.

Composite Detections

Composite detections evaluate a sequence or combination of findings before creating an alert. For example, a suspicious script execution followed by a credential-access event may warrant investigation more readily than either finding alone. This can reduce false positives and redundant work while retaining the underlying events for hunting. The detection needs to retain correlation state and may have to wait for another event. Missing telemetry or a different attack sequence can prevent it from firing, so lower alert volume needs to be checked against detection coverage.

SLA-Based Deferral

An SLA gives us a boundary for how long a job can wait. A low-priority review that can safely wait several hours does not need to compete with an active containment task right now. SLAs should be explicitly defined to guide engineering priorities and manage expectations during a crisis.

Deferral is useful when there is a quieter period in which to finish the work, and the deadline gives us something measurable to enforce. It doesn't remove the work or guarantee a quieter period will arrive. Track age as well as count, retain jobs durably, and decide what happens as deadlines approach.

Admission Control

Attribute-Based Culling

Attribute-based culling excludes work using known context, such as expected activity on a development system or an indicator known to produce benign matches. This can be more precise than raising a threshold across the environment, but the context must justify the exclusion. A development endpoint can still be compromised, and stale ownership or environment tags can suppress a useful alert. Keep exclusions narrow, retain evidence when needed, and measure what gets excluded.

The admission-control scene caps bulk work entering the processing path and retains the excess. A production policy also needs to specify whether excluded work is retained, summarized, or discarded.

Rate Limiting

Rate limiting stops one noisy source from consuming all capacity, especially with per-source or per-tenant limits. The producer needs a policy for excess work: retain it, retry later, or drop it. Immediate retries after a rate-limit response can turn an alert flood into a request flood.

In the backpressure scene, excess work accumulates in finite upstream storage while admission slows. That buffer eventually fills if the flood persists, so rate limits need retention and recovery policies.

Process Work Faster

Processing capacity can increase in two ways:

1. Reduce service time so each worker finishes jobs sooner.
2. Add workers so more jobs can run at once.

Reduce Service Time

Automation

People are resources too. Manual processing adds waiting time, and staffing cannot scale as quickly as an alert flood. Automation reduces that bottleneck by making routine work repeatable.

A playbook can collect endpoint context, check indicators, and prepare a case without an analyst making each API call. It can apply one person's decision across thousands of endpoints, substantially shortening recovery.

Caching and Batching

Caching avoids repeated lookups, such as fetching the same endpoint inventory for hundreds of alerts. Batching groups requests where the API supports it, reducing per-request network and processing overhead.

Cache freshness matters when security state changes quickly, and waiting to fill a batch adds delay. Neither should postpone urgent containment to improve average performance.

Code and Runtime Efficiency

Optimized code and a more efficient runtime reduce the time spent doing the work itself. Removing repeated parsing, unnecessary copies, or an expensive query can matter much more than switching languages. Hot and warm runtimes address a different cost: the time spent starting the execution environment before work can begin.

Some SOAR platforms run native actions efficiently but start a separate runtime for custom code. When I build playbooks, I try to avoid reaching for Python when the native runtime can handle the same task. Starting another execution environment for a small operation can add more latency and resource usage than the operation itself.

Move Work off the Critical Path

Reports and optional enrichment can often wait until after containment. Required approvals remain on the critical path for actions that depend on them. Make clear which steps must finish first and which can run later or be disabled during a flood.

Deferred tasks still need an owner, a deadline, and capacity to finish. Moving work out of the immediate workflow helps only if it can reliably be completed within its SLA.

Add Capacity

Adding workers increases how much work can run at once. Going from one worker to two doubles nominal capacity if per-worker performance stays the same. Actual throughput will depend on whether the jobs can run independently and whether shared databases, APIs, and other dependencies can keep up.

Elastic Scaling

Elastic scaling adds resources as demand increases and removes them when demand falls. It helps with bursts when work can run independently and downstream systems can accept more requests. Scaling takes time, so the queue must hold the work that arrives while new capacity starts. For on-prem or private cloud deployments, the infrastructure team can help establish what can scale, how quickly, and where the limits are.

Managed Capacity with FaaS and SaaS

SaaS hands operation of the service to a vendor, including the infrastructure and runtime behind it. This reduces the amount of infrastructure a security team has to operate. The vendor may add capacity during a burst, but customer quotas and downstream rate limits can still constrain the workload. Those limits need to be understood before an incident.

FaaS, such as AWS Lambda, provides managed execution for individual functions. It can suit short bursts of custom code without requiring the team to operate the execution environment. Startup latency, scaling rate, concurrency quotas, and downstream API limits still apply. Parallel invocation is possible when the work is independent. For frequent or long-running jobs, compare the cost and execution limits with a persistent worker rather than assuming either model is cheaper.

Safe Parallelism

Parallel workers often need to access the same variable, database, or API. Locks can protect shared state, but workers waiting for a lock are not processing other jobs. Keep the protected operation short and check whether one shared resource serializes the whole workflow. In Rust, DashMap distributes map access across shards, which can reduce contention compared with one lock around a HashMap. It still uses locks. Database writes also need to be assessed against the database's concurrency model.

Async and Green Threads

Parallelism runs independent jobs at the same time. Async and green threads overlap I/O waits without dedicating an operating-system thread to each idle task. Since D&R makes many API calls, efficient waiting lets the same resources keep more jobs in flight.

The runtime and its libraries both matter. Python and JavaScript support asynchronous I/O, but are hard to parallelize within a given event loop. Workers or separate processes may be needed for that work. Assess the actual workload and the platform's execution model before choosing a language or assuming that async will solve the bottleneck.

Request Partitioning and Resource Allocation

Partitioning assigns subsets of work to different workers or deployments. That can reduce contention, keep processing near its data, or prevent one tenant's activity from disrupting everyone else. Resource allocation determines how much CPU and memory those workers can use and whether the runtime uses it well.

An uneven partition can leave one worker overwhelmed while others sit idle, and a memory-heavy job can exhaust a deployment before CPU becomes the limit. Measure work distribution and resource consumption. Allocate CPU, memory, and concurrency so critical workflows can run even when a bulk job consumes its full allowance.

Preserve Capacity

Slow dependencies can leave workers waiting for enrichment or retrying failed actions. The dependency-failure scene reduces effective throughput. Protecting important work means controlling access to remaining capacity and limiting which dependencies it needs.

Protect Critical Work

Priority Queues

Priority queues serve important work first, so active containment need not wait behind routine reviews. More delay and loss falls on lower-priority tasks. Priority labels need to reflect response needs, and strict priority can starve other work indefinitely.

Isolated Reserved Capacity

Reserved capacity gives critical work a share bulk work cannot consume. Isolation is essential: separate queues can still share an exhausted API quota or failed dependency. Use separate worker pools, budgets, or workflows where needed. The example reserves 25% of capacity for critical jobs; bulk work cannot borrow it, so some capacity may sit idle.

Reduce Dependency Risk

Circuit Breakers / Dependency Bypass

A circuit breaker stops repeatedly calling a dependency that is failing. After enough failures, the caller opens the circuit and returns quickly, freeing resources that would otherwise wait for timeouts or retries. A limited probe later checks whether the dependency has recovered. The workflow needs a defined response while the circuit is open, such as deferring the job or using a fallback.

A fallback lets a workflow continue when a dependency is unavailable. For example, if a SOAR's TIP integration becomes slow or starts returning errors, a playbook could query a threat intel provider directly. Some response actions may also be available through both an EDR platform and an MDM solution, such as Jamf or Intune. Those alternatives need to be configured and tested before they are needed. Jobs blocked on one failed dependency should not occupy every worker while other response tasks wait.

Minimum Viable Operation

Security teams should define their minimum viable operation: the steps and dependencies needed to continue useful response work. Collecting essential evidence, escalating an active compromise, and executing known containment procedures may take precedence over optional lookups or detailed reporting. Document the reduced operating mode, the evidence it uses, the actions it permits, and the follow-up work it creates. Then test that those essential tasks still complete when optional services are unavailable.

Bound the Damage

A prolonged flood can exceed processing capacity and storage. Decide what fails first, or whichever buffer fills first may discard critical evidence while retaining work that could wait.

Define the Failure Order

Decide what must survive, what can degrade, what can wait, and what can be discarded. Apply those decisions to both security tasks and the data that supports them. Security engineering, senior leadership, GRC, legal, and other teams responsible for business continuity need to be involved.

For each workload, name the retention requirements, degradation trigger, and recovery owner. Losing optional enrichment differs from losing the underlying event. Test the failure order: an optional step that blocks a critical workflow is still a dependency.

Control Incoming Work

Upstream Restrictions and Backpressure

Backpressure tells an upstream system to slow delivery. Pull consumers can stop fetching; push integrations may need retry responses or producer-side limits. Establish where excess work is retained and for how long. If the source cannot slow down or retain it, work is shed. A controlled local queue can hide a growing upstream backlog.

Failure Detection in Push and Pull Queues

Failed push delivery can leave gaps after both systems recover. I like a push/pull approach: push by default, then check for gaps and pull missing records where the source supports it.

Compare upstream records with what reached the end of the workflow, usually the SOAR. Retrieve missing records from the last confirmed point and alert the integration owners. Rate limits and duplicate handling keep recovery from creating another flood or repeating response actions.

Retain or Shed Excess Work

Durable Deferral

Deferred jobs need durable storage, original timestamps, and enough context for later processing. Track age and storage usage, distinguishing current inventory from cumulative deferrals. The scene retains excess jobs outside the waiting queue. That store is finite, so processing capacity still needs to recover.

Explicit Load Shedding

Load shedding discards work deliberately to keep the remaining system available. It should follow the failure order rather than simply dropping whichever event arrived last. We may choose to discard repeat notifications while retaining original evidence, or stop creating certain low-value jobs during an emergency.

Record losses by amount, source, and reason, and make them visible. Low severity alone does not justify discarding evidence. In the scene, prioritizing bulk overflow limits damage but cannot recover lost work.

Plan Recovery

Recovery Takes Capacity

Recovery must handle new arrivals while draining the backlog. Spare capacity is processing rate minus arrival rate. If normal work consumes nearly all capacity, recovery can outlast the incident. Monitor job age and keep reduced operation in place until the normal workload is safe.

Controlled Replay

I like copying alerts into a separate store using a tool like Cribl, then replaying them through existing playbooks. Budget for storage and retention, limit replay to spare capacity, and preserve timestamps. Check whether an old event still warrants the same response action.