Recommended CAST AI Workload Autoscaler Scaling Policy Settings by Workload Type

Audience: platform and infrastructure engineers running Kubernetes
Scope: this post covers detailed Scaling Policy settings for the CAST AI Workload Autoscaler (WOOP). Node autoscaling and the Evictor are out of scope, and native Kubernetes settings (PDBs, probes, etc.) are only mentioned as supporting tips.

Before we start: system policies and automatic assignment

When you enable the Workload Autoscaler, CAST AI automatically profiles every workload and assigns it to a matching system policy based on labels, workload type, and namespace. Basic optimization kicks in right after onboarding without any manual setup.

The system policies are:

System policy	Character	Default assignment
balanced	Balances cost and stability. The safe default	Default for everything except StatefulSets
resiliency	Stability first, conservative about changes	Default for StatefulSets
stability	Consistent performance, minimal disruption	Performance-sensitive workloads
cost-savings	Maximizes cost reduction	Savings-first workloads
burstable	Scales resources up when needed, then reclaims them. 7-day look-back	Jobs, CronJobs, and other job-like workloads
readonly	Reserved for CAST AI’s own components. Cannot be modified	castware

In other words, even if you do nothing, workloads get matched to a reasonably appropriate policy on their own. So why does this post exist?

Because system policies are a solid starting point, not the right answer for your specific workloads. They can’t be modified directly (only duplicated), and they don’t account for the finer details: JVM startup spikes, the look-back requirements of periodic traffic, confidence issues with batch jobs, and so on. For the detailed workload cases covered in this post, we recommend creating a Custom Policy via Create Scaling Policy. Start with the system policies, then peel off groups of workloads with distinct characteristics into custom policies one at a time. That’s the backbone of this post.

We’ll classify workloads into eight types, explain why each type behaves the way it does from a scaling perspective, lay out recommended Custom Policy settings for each, and finish with Terraform and API examples.

---

0. Scaling Policy settings you should know first

Defaults vary by system policy. In the table below, only the balanced policy values are shown; everything else is marked “varies by policy.”

Setting	What it means	Default
Optimization mode	Set to Read-only (recommendations only) or Vertical (applied) in the console. In the API/Terraform these map to READ_ONLY / MANAGED	Varies by policy
applyType	IMMEDIATE (applied right away via pod restart or in-place) / DEFERRED (applied on natural restarts; K8s 1.33+ attempts in-place resize)	balanced: IMMEDIATE
Recommendation percentile	Target usage percentile (QUANTILE + p-value / MAX), set separately for CPU and memory	balanced: CPU p80, Memory max
Overhead	Buffer added on top of the recommendation	balanced: CPU 10%, Memory 10%
Change sensitivity	How much a recommendation must differ from the current request before it’s applied (PERCENTAGE fixed / DEFAULT_ADAPTIVE dynamic)	Varies by policy (Dynamic recommended)
Look-back period	Historical data window for recommendations (3 hours to 7 days, separate for CPU/memory)	balanced: 3 days (burstable: 7 days)
Constraints (min/max)	Upper and lower scaling bounds (absolute values or % of original request)	None
Limits handling	Remove / keep / preserve ratio / multiplier	Memory: automatic 1.5x
Startup metrics	How the startup window is handled: include / startup recommendations (original requests during startup, optimized values after) / ignore	System policies: first 2 minutes ignored
Confidence	Minimum data confidence required for automatic application	90%
Rollout behavior	Zero-downtime (temporarily 2 replicas for single-replica workloads), one-by-one eviction, eviction delay	Opt-in
Stall detection	Raises recommendations when PSI-based CPU pressure is detected (stall %, pod %)	Varies by policy (balanced: 10%)
Memory event	How to apply on memory events such as OOM	Varies by policy
Downscaling	Separate apply type for downscaling only	Varies by policy
JVM optimization	Memory recommendations based on JVM heap metrics	Off
Excluded containers	Containers excluded from optimization (e.g. sidecars)	None
Horizontal autoscaling	Policy-level creation and management of native HPA (v2): replica range, triggers, behavior, take-ownership	Off
Assignment rules	Automatic assignment by namespace / GVK / label expressions	Varies by policy

Two important interactions:

1. Only enable policy-level Horizontal autoscaling if your environment has no HPA of its own. When vertical and horizontal are both on, CAST AI automatically balances the two dimensions (and the CPU overhead setting is ignored), but if your team already runs its own HPA or KEDA, you end up with two owners and a lot of confusion.
2. If a workload already has its own HPA (self-managed, KEDA, etc.), applying vertical optimization is nothing to worry about. The Workload Autoscaler is designed to recognize the existing HPA and work alongside it, so you can leave your HPA in place, turn on vertical only, and things just work.

Two operating principles:

• Always start a new custom policy in Read-only, watch the recommendations for a week or two, then switch it to Vertical.
• Turn on Zero-downtime updates in every policy except the StatefulSet one. Most environments have more single-replica Deployments than people expect, and without this setting an IMMEDIATE apply becomes straight downtime. Even on K8s 1.33+ where in-place updates work, there are changes in-place can’t handle, so we recommend keeping it on regardless.

One more note: if you run an auto-syncing CD tool like ArgoCD, there’s nothing to worry about. The resource values the Workload Autoscaler applies aren’t edits to the Deployment manifest; they’re injected into Pods at creation time via a webhook, so there’s no risk of conflict with GitOps sync (diff/self-heal).

---

1. Stateless services – web frontends / APIs / mobile backends / BFF / gRPC

What makes these workloads tick

Stateless services keep no local state in the pod. Data lives only for the duration of a request, and anything session- or cache-like is pushed out to external stores (Redis, a database). From a scaling perspective, this is the good case, for three reasons.

First, restarts are easy. If a pod dies, another replica picks up the requests immediately, and a new pod can take traffic within seconds. That means WOOP can restart pods to apply recommendations with essentially no service impact, which is why IMMEDIATE is safe here.

Second, resource usage tracks traffic closely. CPU rises and falls with RPS, and memory stays fairly stable, driven by connection counts and buffer sizes. Past usage genuinely predicts future usage, which is exactly the profile percentile-based recommendations are best at.

Third, horizontal scaling actually works. Add replicas and the load balancer spreads the traffic. This is the one category where you can run vertical (request optimization) and horizontal (replica count) together.

There are sub-type caveats, though. gRPC reuses HTTP/2 connections, so behind an L4 load balancer, new pods may never receive traffic – a connection imbalance where scaling out just leaves the old pods busy. L7 balancing or client-side load balancing is a prerequisite. WebSocket and other real-time services hold connections open for minutes to hours, so taking a pod down triggers a reconnection stampede. The “restarts are easy” assumption weakens here, and rollouts should be slowed down.

Recommended settings

Setting	Recommendation	Why
applyType	IMMEDIATE	Restarts are cheap, so optimize fast
Zero-downtime updates	On	Protects single-replica Deployments. Still essential on 1.33+
CPU percentile	QUANTILE p80	Momentary peaks are absorbed by bursting
Memory	MAX + 10-15% overhead	Memory is incompressible; stay conservative
Sensitivity	DEFAULT_ADAPTIVE	Works well for a mixed bag of service sizes
Look-back	1-2 days	Captures daily patterns while staying responsive
Rollout	One-by-one + 10-60s eviction delay	For long-lived connections (gRPC/WebSocket), increase the delay to allow draining
Horizontal	If you have no HPA of your own: min 2, CPU trigger at 70-80%, scale-down stabilization 300s	Availability floor + flapping prevention
CPU limit	Remove limits	Prevents throttling

Kubernetes tip: combining terminationGracePeriodSeconds and a preStop hook for connection draining, plus a PDB minAvailable, gives you noticeably better service quality.

2. Services with volatile traffic – spiky / periodic surges

What makes these workloads tick

The essence of this category is that the gap between average and peak is large. If a service averages 200m CPU but burns 2000m at lunchtime, rightsizing toward the average guarantees an incident at every peak. This is the trickiest profile for any autoscaling tool, and the strategy splits on whether the volatility is predictable.

Periodic patterns (commute hours, lunchtime, weekends, end-of-month settlement) look regular as a time series. The trouble starts when the look-back window is shorter than the cycle. Say look-back is 24 hours and traffic drops to a tenth on weekends: on Sunday evening WOOP learns “this workload barely uses anything” and slashes the requests. Then Monday 9 a.m. arrives and everything falls over. This is where the principle comes from: the look-back must cover the full peak cycle. The regularity is also an opportunity – predictive scaling can learn the pattern and raise resources before demand arrives.

Unpredictable spikes (event launches, ticket sales, news-driven traffic) are the type where history tells you nothing about the future. The reactive scaling chain (metric collection => HPA decision => pod startup => node provisioning if needed) takes minutes, and by then the spike has either passed or caused an outage. So for this type, WOOP’s job isn’t “optimization” – it’s making sure everyday rightsizing doesn’t eat into your spike headroom. You counter the downward pressure from low-traffic history with higher percentiles and min constraints.

2-a. Periodic patterns – recommended settings

Setting	Recommendation	Why
Look-back	7 days	Must include the weekday/weekend cycle so Monday doesn’t blow up after a weekend downscale
Predictive scaling (CPU)	Enabled	Learns the cycle and raises resources ahead of demand
CPU percentile	p90-p95	Absorbs the periodic peaks
Zero-downtime updates	On	Protects single replicas

2-b. Unpredictable spikes – recommended settings

Setting	Recommendation	Why
Percentile	p99 to MAX	Average-based recommendations are meaningless for spikes
Memory overhead	20-30%	Prevents OOM on the first spike
Predictive scaling (CPU)	Enabled	Catches whatever repeating patterns are hiding inside the spikes
Constraints min	A generous absolute value (or 90-100% of the original request)	A hard floor against over-shrinking based on quiet periods
Stall detection	Stall threshold 10%	Early detection of hidden CPU contention
Zero-downtime updates	On	Protects single replicas
Horizontal	If you have no HPA of your own: generous max replicas, scale-up stabilization 0s	Horizontal scaling is the primary defense against spikes

Kubernetes tip: the real spike defense is node headroom and HPA behavior. Think of WOOP as the thing that keeps everyday rightsizing from eroding your spike readiness.

3. JVM and warm-up-heavy services – Java / .NET / cache warming

What makes these workloads tick

JVM workloads fool the autoscaler twice.

The first is the startup window. For several minutes after boot – class loading, Spring context initialization, JIT compilation – CPU usage spikes to 2-5x the steady state. If those samples get mixed into the usage history, percentile-based recommendations inflate, requests stay far above what steady state needs, and money is wasted. But if you ignore startup metrics and size for steady state instead, startup crawls on starved CPU, readiness drags, and rolling deployments stall. Generous at startup, tight at steady state. Separating the two phases is the only correct answer, and it’s exactly why startup recommendations exist.

The second is memory. The JVM claims its heap up front per -Xmx and lets the GC manage memory inside it. The memory usage visible outside the container is “what the JVM took from the OS,” not “what the application actually uses.” With a 4Gi heap, container metrics look about the same whether live objects total 500Mi or 3Gi, so container-metric-based rightsizing is structurally inaccurate for the JVM. That’s why CAST AI’s JVM optimization looks at actual heap usage metrics instead of container memory. There’s also one brutal constraint: if the request gets scaled below heap + metaspace + off-heap (native buffers, thread stacks), the JVM doesn’t shrink gracefully – it gets OOMKilled on the spot. That’s why a min constraint here is mandatory, not optional.

.NET has the same structure (JIT + GC + managed heap), so everything above applies. Services that only perform well after a cache warm-up also fit this policy, since they share the same “separate the startup phase” requirement.

Recommended settings

Setting	Recommendation	Why
Startup metrics	Startup recommendations mode, period 5-10 min	Original CPU requests during startup => optimized values after. Solves recommendation inflation and HPA false alarms at once
JVM metrics collection + Auto instrument	Turn both on	With Custom metric JVM collection and Auto instrument enabled together, recommendations are computed from real heap usage. The single highest-impact setting for JVM
Memory	MAX + 15-20% overhead	Absorbs heap + metaspace + off-heap variation
Constraints min (memory)	An absolute value that covers -Xmx + off-heap	The hard guardrail against instant death from scaling below the heap
Zero-downtime updates	On	Protects single replicas
Rollout	IMMEDIATE + one-by-one + eviction delay 60s or more	Warm-up is slow, so never restart everything at once

⚠️ Check this before enabling JVM metrics collection and Auto instrument. The feature exposes and collects JVM metrics over JMX. If another APM or monitoring agent that uses a -jmx input – Datadog, for example – is already attached, the JMX ports can collide. Verify the existing agent’s JMX port configuration before rollout, and separate the ports if they conflict.

Kubernetes tip: use a startupProbe to keep traffic away until warm-up completes. If you use -XX:MaxRAMPercentage, verify that the heap actually moves along with WOOP’s request changes.

4. Stateful and data workloads – Elasticsearch / Redis / Kafka / in-cluster DB

What makes these workloads tick

This category is the exact opposite of the previous three. The pods hold data, and the cost of a restart or relocation is far higher than whatever you’d save on resources.

Here’s what “expensive” looks like in practice. When an Elasticsearch data node restarts, its shards go unassigned and recovery triggers heavy shard-rebalancing traffic. If each node holds hundreds of gigabytes, recovery takes tens of minutes to hours, and the cluster limps along in yellow the whole time. When a Kafka broker goes down, leadership for its partitions moves, and ISR re-synchronization follows once it comes back. If Redis is a primary store, a restart means either a cache-miss storm or, depending on persistence settings, real data-loss risk. An in-cluster RDBMS cascades through connection draining, replication resync, and in the worst case a failover.

On top of that, these workloads are pinned to specific AZs by their PVs, and their memory patterns are unusual. Databases are designed to use every available byte for page cache and buffer pools, so “usage looks high” is intended behavior, not waste. Shrink based on usage metrics alone and the cache hit rate quietly degrades along with performance.

So the strategy for this category isn’t “optimize fast” but observe without disruption, then apply selectively. Keep receiving recommendations in Read-only to size up how over-provisioned things are, then apply via DEFERRED – piggybacking on natural restarts (version upgrades, maintenance) or letting K8s 1.33+ in-place resize handle it. The one exception is volatile cache Redis. If losing the data just means refilling from the source, it can be treated almost like a stateless service. Same Redis, different policy, depending on what it’s for.

Recommended settings

Setting	Recommendation	Why
Optimization mode	Read-only at first	Review recommendations, apply manually; move a subset to Vertical once trust is built
applyType	DEFERRED	No pod restarts. In-place on 1.33+, otherwise applied on natural restarts
Memory	MAX + 20-35% overhead	An OOM on a data workload is an incident
CPU percentile	p95 or higher	Accounts for compaction and rebalancing peaks
Sensitivity	PERCENTAGE 20%	Blocks frequent changes
Look-back	7 days	Covers periodic work like backups and rebalancing
Anti-affinity	consider_anti_affinity = true	Respects constraints during placement
Zero-downtime updates	Off	Not supported for StatefulSet/PVC workloads (the only category-level exception)

Split Redis by purpose: a volatile cache can lean toward category 1 and even run IMMEDIATE; a primary datastore follows this category. Split the assignment rules by label (cache-tier=volatile) and you get two clean policies.

Kubernetes tip: set a PDB with maxUnavailable: 1, and for Elasticsearch, split policies by node role (master = Read-only, coordinating = relaxed) for safer operation.

5. Batch and Jobs – CronJob / big data / DB jobs / ML training / CI runners

What makes these workloads tick

Batch has a different lifecycle. A service stays up while its load varies; a job lives a finite life of start => process => exit. That difference changes two assumptions behind autoscaling.

First, “restart to apply a new request” doesn’t exist as a concept. Killing a running job isn’t optimization, it’s lost work. Instead, the next run is the natural moment to apply. Recommendations are built from this run’s observed usage and picked up when the next scheduled run starts, which makes DEFERRED the natural fit.

Second, resource usage moves in phases. A typical ETL job goes data load (memory surge) => transform (CPU heavy) => write (I/O wait), with the bottleneck shifting per phase and the peak concentrated in one of them. Shave that peak with a percentile and the next run OOMs at exactly that phase; then, depending on the job’s retry policy, you enter a loop of OOM => retry => OOM at the same spot. That’s why batch memory should be sized at MAX, and why memory_event = IMMEDIATE matters – it raises memory on the spot and breaks the loop.

Third, metric density is low. A job that runs 30 minutes once a day leaves 30 minutes of data out of 24 hours. At the default 90% confidence, the system keeps deciding “not enough data” and optimization may never start. The less frequently a job runs, the more you should lower confidence and stretch the look-back to match the cycle.

The sub-types have different temperatures, too. CI runners live for minutes and are cheap to re-run, so they can be optimized most aggressively. Big data and ML training run for hours with expensive mid-run losses, so it comes down to whether checkpointing exists. DB jobs (migrations, backups) may be short, but an interruption can break data consistency, so they warrant protection on par with category 4.

Recommended settings

Setting	Recommendation	Why
applyType	DEFERRED	Applies on the next run; leaves running jobs alone
Look-back	Cover the full job cycle (daily = 2 days, weekly = 7 days)	If the peak falls outside the window, the next run OOMs
Memory	MAX + 20-35% overhead	Batch memory peaks are brief and lethal
CPU percentile	p95 to max	A batch job using all its CPU is normal
Memory event	IMMEDIATE	Immediate raise on OOM breaks the retry loop
Confidence	Relax to 70-80%	Low run frequency means low data density. At 90%, optimization may never start

DB jobs sit at the intersection of categories 4 and 5: DEFERRED plus a generous memory min, and if an interruption could break consistency, keep them in Read-only and use the recommendations as reference only.

6. GPU – inference / training

What makes these workloads tick

GPU workloads earn their own category for three reasons. The costs are an order of magnitude higher (GPU nodes run several times to tens of times the price of regular nodes), GPUs don’t subdivide (whole units per pod unless you use time-slicing/MIG), and nodes are slow to obtain (GPU instances have limited availability and can take minutes or more to provision).

One point matters most here: what WOOP directly optimizes is not the GPU but the GPU pod’s CPU and memory requests. GPU pods still use CPU and memory for preprocessing, tokenization, and data loading, and if those values are inflated, nothing else can fit into the leftover space on an expensive GPU node. Getting CPU/memory requests right on GPU nodes pays off far more than on regular nodes, simply because the nodes being bin-packed cost so much.

Operationally there are two shapes. Inference is request/response serving, so it behaves like a stateless service – except for a warm-up phase where a multi-gigabyte model gets loaded into GPU memory at startup. That’s the same structure as category 3 (JVM/warm-up), so set the startup recommendations period to the model load time. For horizontal scaling, GPU utilization or concurrent request count is the accurate signal, not CPU utilization. Training is a batch job that runs for hours to days. A restart erases all computation since the last checkpoint, so IMMEDIATE, which can trigger restarts, is off the table; apply the category 5 batch policy.

Recommended settings

• Inference (serving): IMMEDIATE + Zero-downtime updates on + startup recommendations (period = model load time) + one-by-one. GPU-utilization-based horizontal scaling belongs to the HPA / custom metrics layer.
• Training: DEFERRED + look-back covering the training cycle + memory MAX. To protect checkpoints, do not use IMMEDIATE.

7. Infrastructure and system components – DaemonSet / Istio / Argo / monitoring

What makes these workloads tick

This category is less “things to scale” and more components that ride as overhead on every other workload in the cluster. Each pod looks small, but the multiplier is large – that’s the common thread.

A DaemonSet has no adjustable replica count; the node count is the pod count. Horizontal scaling is conceptually impossible, so only vertical applies. But the leverage is big: trim a DaemonSet request by 100m and you get node count x 100m back in one move. The flip side is that a single recommendation change means restarts on every node, so change frequency should be minimized. DaemonSet requests are also a fixed overhead pre-claimed on every node – if they’re inflated, bin-packing efficiency suffers cluster-wide.

With the Istio sidecar (istio-proxy), the multiplication happens per pod. It’s injected into every pod in the mesh, so if the sidecar’s default request (say CPU 100m / Mem 128Mi) exceeds actual usage (a few dozen millicores), the cluster wastes that difference times the total pod count. Including the sidecar in optimization adjusts app container and sidecar together, so the recommended approach is to start with it in excludedContainers, observe, and then include it in stages.

Argo Rollouts is a deployment controller that manipulates ReplicaSets itself to run canary/blue-green. If a WOOP restart lands in the middle of an in-flight rollout, the analysis phase can be contaminated. In deploy-heavy environments, set DEFERRED and let the deployments themselves be the apply moments – it’s cleaner. Argo Workflows shares only the name; its nature is batch, so assign it to the category 5 policy. And as mentioned earlier, an auto-syncing CD like ArgoCD is not a problem: recommendations are injected into Pods via webhook, not written to the Deployment, so there’s no conflict with GitOps sync settings.

Monitoring stacks (Prometheus and friends) have memory that grows in steps with metric cardinality (the number of time series), not with traffic. A new service rollout or an added label bumps the series count, memory steps up, and it doesn’t come back down – so MAX with a generous overhead is the safe choice.

Recommended settings

• DaemonSet: DEFERRED + memory MAX + sensitivity 20% (one change = restarts across every node). Stall detection does not apply to DaemonSets (documented limitation).
• Istio sidecar: start with excludedContainers: [istio-proxy] (conservative), then include the sidecar (aggressive) after reviewing the metrics.
• Argo Rollouts: one-by-one rollout is officially supported. If deployments are frequent, DEFERRED is recommended.
• Argo Workflows: assign to the category 5 batch policy.
• Prometheus and other monitoring: memory MAX + 30% overhead + DEFERRED. One caution: on K8s 1.33+ with in-place resizing, the recommendation-applied-at annotation can refresh every 30 minutes, and collectors that index annotations as labels can suffer a cardinality explosion. Exclude it from collection.

8. Workloads that are already in trouble – OOM and anti-patterns

Existing problems can be fixed with the Workload Autoscaler too

The items here aren’t workload types; they’re conditions already causing trouble in your cluster. The good news is that workloads with pre-existing problems can also be fixed through the Workload Autoscaler. Chronic conditions – the workload that OOMs on repeat, the single-replica service everyone is afraid to restart – become manageable with a handful of policy settings.

Start by understanding how OOM works. The moment a container crosses its memory limit, the kernel kills it. There’s no riding it out with throttling the way CPU does. Rightsizing is, by nature, the act of lowering requests toward observed usage – so any peak that the look-back window didn’t capture leads straight to an OOM. The defense is three layers deep: (1) a look-back longer than the peak cycle (don’t miss the peak), (2) overhead of 20% or more (absorb unobserved variation), and (3) memory_event = IMMEDIATE (if it still blows, raise memory instantly and stop the repeat).

There’s one more case worth calling out. If the OOM is caused by a memory leak, the story changes. On a leaking workload, memory_event keeps raising the recommendation after every OOM, and you can end up with unexpectedly enormous nodes being provisioned over and over. Limit settings alone won’t contain this. To prevent it, set an explicit constraint memory max at a value the workload could never legitimately reach (for example, 30Gi). As a bonus, a recommendation pinned against that ceiling is itself a strong signal you have a leak.

Checklist

• OOM: memory_event = IMMEDIATE + memory limit Automatic (1.5x) + overhead 20% or more. Keep the look-back longer than the peak cycle, always.
• Suspected memory leak: set constraint memory max to an unreachable value (e.g. 30Gi) so a leak can’t snowball into ever-larger node provisioning.
• Single replica with no PDB: an IMMEDIATE restart is downtime, full stop. Turn on Zero-downtime updates (temporary second replica). Deployment-only, and no PVCs.
• CPU limits everywhere: the classic cause of throttling. Removing limits is the baseline move.
• Every workload in one policy: percentile, look-back, and applyType are all functions of workload character. Workloads with different characters belong in separate policies.

---

Summary table

Category	applyType	CPU	Memory	Look-back	Key switches
1. Stateless	IMMEDIATE	p80	MAX +10%	1-2d	Zero-downtime, one-by-one
2-a. Periodic traffic	IMMEDIATE	p90-95	MAX	7d	Predictive scaling
2-b. Spikes	IMMEDIATE	p99-MAX	MAX +20-30%	7d	Min constraint, predictive, stall detection
3. JVM / warm-up	IMMEDIATE	p80	MAX +15-20%	2d	JVM metrics + Auto instrument, startup recs
4. Stateful	DEFERRED	p95	MAX +20-35%	7d	Start Read-only, sensitivity 20%
5. Batch / Jobs	DEFERRED	p95-MAX	MAX +20-35%	Job cycle	memory_event IMMEDIATE, relaxed confidence
6. GPU	Inference IMMEDIATE / training DEFERRED	Per categories 1, 3 / 5			Startup period = model load
7. Infrastructure	DEFERRED	p90	MAX +30%	7d	excludedContainers (istio-proxy)
8. (Cross-cutting)	–	–	–	Longer than peak cycle	Constraint max (leak defense), zero-downtime

---

Codifying it with Terraform

The castai_workload_scaling_policy resource in the castai/castai provider lets you manage all of this as IaC. Here are three representative policies.

# 1. Stateless policy
resource "castai_workload_scaling_policy" "stateless" {
  name              = "stateless-services"
  cluster_id        = castai_eks_cluster.this.id
  apply_type        = "IMMEDIATE"
  management_option = "MANAGED" # Maps to Vertical in the console. Start with READ_ONLY (console: Read-only) on first rollout

  assignment_rules {
    rules {
      workload {
        gvk = ["Deployment"]
        labels_expressions {
          key      = "tier"
          operator = "In"
          values   = ["frontend", "api", "bff"]
        }
      }
    }
  }

  cpu {
    function                 = "QUANTILE"
    args                     = ["0.80"]
    overhead                 = 0.10
    look_back_period_seconds = 172800 # 2d
    apply_threshold_strategy { type = "DEFAULT_ADAPTIVE" }
  }

  memory {
    function                 = "MAX"
    overhead                 = 0.10
    look_back_period_seconds = 172800
    apply_threshold_strategy { type = "DEFAULT_ADAPTIVE" }
    limit {
      type       = "MULTIPLIER"
      multiplier = 1.5
    }
  }

  rollout_behavior { type = "NO_DISRUPTION" } # one-by-one sequential eviction

  confidence { threshold = 0.9 }
}

# 3. JVM policy
resource "castai_workload_scaling_policy" "jvm" {
  name              = "jvm-services"
  cluster_id        = castai_eks_cluster.this.id
  apply_type        = "IMMEDIATE"
  management_option = "MANAGED"

  assignment_rules {
    rules {
      workload {
        gvk = ["Deployment", "StatefulSet"]
        labels_expressions {
          key      = "runtime"
          operator = "In"
          values   = ["jvm"]
        }
      }
    }
  }

  cpu {
    function                 = "QUANTILE"
    args                     = ["0.80"]
    overhead                 = 0.10
    look_back_period_seconds = 172800
    apply_threshold_strategy { type = "DEFAULT_ADAPTIVE" }
  }

  memory {
    function = "MAX"
    overhead = 0.20
    min      = 1.0 # GiB floor - guardrail for heap + off-heap
    apply_threshold_strategy { type = "DEFAULT_ADAPTIVE" }
  }

  # Memory recommendations from JVM heap metrics
  jvm {
    memory { optimization = true }
  }

  # Startup window: keep the spike from inflating recommendations
  startup {
    period_seconds = 420 # 7min
  }

  rollout_behavior { type = "NO_DISRUPTION" }
}

# 4. Stateful policy
resource "castai_workload_scaling_policy" "stateful" {
  name              = "stateful-data"
  cluster_id        = castai_eks_cluster.this.id
  apply_type        = "DEFERRED"  # no pod restarts
  management_option = "READ_ONLY" # recommendations only (console: Read-only) - switch to Vertical after validation

  assignment_rules {
    rules {
      workload { gvk = ["StatefulSet"] }
    }
    rules {
      namespace { names = ["elasticsearch", "kafka", "redis-primary"] }
    }
  }

  cpu {
    function                 = "QUANTILE"
    args                     = ["0.95"]
    overhead                 = 0.15
    look_back_period_seconds = 604800 # 7d
    apply_threshold_strategy {
      type       = "PERCENTAGE"
      percentage = 0.2 # minimize change frequency
    }
  }

  memory {
    function                 = "MAX"
    overhead                 = 0.30
    look_back_period_seconds = 604800
    apply_threshold_strategy {
      type       = "PERCENTAGE"
      percentage = 0.2
    }
  }

  anti_affinity { consider_anti_affinity = true }
  downscaling   { apply_type = "DEFERRED" }
  memory_event  { apply_type = "IMMEDIATE" } # on OOM, raise immediately
}

Policy-level native HPA settings (hpa_settings: replica range, triggers, behavior, take-ownership) and GPU management fields were added in recent provider versions. Check your provider version and the schema in the resource documentation before using them.

Creating policies via the API

Endpoint: POST /v1/workload-autoscaling/clusters/{clusterId}/policies

curl --request POST 
  --url "https://api.cast.ai/v1/workload-autoscaling/clusters/${CLUSTER_ID}/policies" 
  --header "X-API-Key: ${CASTAI_API_KEY}" 
  --header "Content-Type: application/json" 
  --data '{
    "name": "spike-services",
    "applyType": "IMMEDIATE",
    "cpu":    { "target": "p99", "overhead": 0.20 },
    "memory": { "target": "max", "overhead": 0.30 },
    "assignmentRules": [
      {
        "workload": {
          "gvk": ["Deployment"],
          "labelsExpressions": [
            {
              "key": "traffic-pattern",
              "operator": "KUBERNETES_LABEL_SELECTOR_OP_IN",
              "values": ["spike", "event"]
            }
          ]
        }
      }
    ]
  }'

Use GET .../policies and PUT .../policies/{policyId} for listing and updates. You can trace how each workload was assigned via the scalingPolicyOrigin field (api / annotations / assignment-rules / default) in the workload API response. See the API reference for the full schema.

Per-workload exceptions are handled with annotations, which take top priority.

metadata:
  annotations:
    workloads.cast.ai/configuration: |
      scalingPolicyName: stateless-services
      vertical:
        optimization: on
        excludedContainers:
          - istio-proxy

---

Closing thoughts

It boils down to two sentences.

1. System policies are the starting point; custom policies are the destination. Begin with automatic assignment, then peel off workload groups with distinct characteristics into their own policies via Create Scaling Policy.
2. The look-back must always be longer than the workload’s peak cycle. Seven days for periodic traffic, the job cycle for batch, and 1-2 days for everything else.

References

• Scaling Policies, Settings Reference, Annotations Reference
• Terraform: castai_workload_scaling_policy
• Workload Optimization API