How Do I Keep a Shared CPU Fleet Stable When Workloads Vary by Day?

Managing a fleet of shared CPU instances in the cloud is a familiar challenge for infrastructure engineers and Site Reliability Engineers (SREs). When workloads fluctuate—sometimes dramatically—day-to-day, ensuring stable performance without overspending demands a thoughtful approach.

In this article, we’ll explore practical strategies for maintaining stability in shared CPU fleets across cloud providers, leveraging tools like AWS Compute Optimizer and Azure Advisor. Along the way, we’ll debunk common myths such as treating vCPU counts as hard performance guarantees and relying on average CPU utilization for cost and scaling decisions.
Understanding Workload Variability
Before jumping into optimizations, it’s critical to understand the shape and variability of your workloads. Many engineers default to looking at average CPU utilization to make instance type and autoscaling decisions. This approach, however, often leads to oversizing or undersizing because averages hide spikes—precisely when users will notice degraded service.
Measure Peaks with the Right Observation Window
Cloud workloads often have complex temporal patterns:
Diurnal spikes (day/night differences) Weekly cycles (business days vs weekends) Batch processing or cron jobs causing short bursts
It’s crucial to select an appropriate observation window that captures your workload’s real peak usage:
Avoid overly short windows: Spotting short CPU spikes lasting seconds may not justify larger instances if those spikes don’t affect SLAs. Avoid overly long windows: Averaging behavior over weeks will mask daily peaks and cause misinterpretation.
Typical working windows are hourly or 5-minute metrics over at least 1–2 weeks to reveal habitual peak patterns.
Use Percentiles and Spike Duration, Not Averages
Percentiles such as the 95th (P95) and 99th (P99) CPU utilization levels provide much more actionable insight:
P95 utilization: Gives an idea of the load under which your fleet performs acceptable most of the time. P99 utilization: Shows extreme spikes that need mitigation tactics like autoscaling or burst capacity.
Equally important is quantifying how long these spikes last. For example, a P99 CPU spike that lasts 10 seconds and affects no user experience may be tolerable; sustained P99 usage over several minutes is a clear sign of underprovisioning.
What Does Shared CPU Mean in Practice?
Shared CPU instances (e.g., AWS T-series or Azure B-series) are deceptively named. The nature of “sharing” varies significantly between cloud providers and must be understood to develop stable fleet management strategies.
AWS Shared CPU Instances
AWS offers 'burstable' T-series instances that accumulate CPU credits when idle, allowing short bursts of full-core utilization. Key points include:
CPU credits can be banked and spent, allowing bursts beyond baseline. CPU credit depletion leads to throttled CPU performance. There is no strict vCPU-to-core performance guarantee; a vCPU represents a hyper-threaded thread. Azure Shared CPU Instances
Azure B-series applies a similar CPU credit concept with differences:
CPU credits replenish slowly with baseline CPU assigned to the instance size. Credit pooling can happen at a subscription or VM level depending on instance family. Workloads sensitive to latency or compute bursts may observe different behaviors than AWS.
Understanding these differences informs scheduling, autoscaling, and capacity planning.
Strategies to Keep Your Shared CPU Fleet Stable 1. Avoid Always-On Small Services That Hide Waste
One common anti-pattern is maintaining a large number of always-on tiny instances to handle dynamic workloads. While "small" instances seem cheap, they often cost more cumulatively and increase operational complexity.

Instead:
Use autoscaling (horizontal and vertical) to match capacity to actual demand peaks. Consolidate small workloads where feasible to reduce fixed overhead. Measure and optimize egress and storage costs in tandem with compute. 2. Use AWS Compute Optimizer and Azure Advisor to Inform Decisions
Both AWS Compute Optimizer and Azure Advisor analyze your historical usage and recommend right-sizing and savings opportunities.
Tool Cloud Provider Key Features AWS Compute Optimizer AWS Analyzes CPU, memory, network usage over weeks Provides recommendations for instance type and family changes Supports Auto Scaling Group right-sizing Azure Advisor Azure Identifies underutilized resources Suggests VM resizing within and across families Recommends autoscaling and purchasing options
I'll be honest with you: these automated insights complement your detailed percentile-based analysis and help avoid hand-wavy cost or performance estimates.
3. Implement Autoscaling With Spike Duration Awareness
Autoscaling policies that trigger on average CPU utilization can be misleading—especially for shared CPU instances where transient spikes are common.

Best practices include:
Use target tracking with P95 or P99 CPU utilization metrics if possible. Incorporate cooldown periods matching your workload spike durations. Combine horizontal autoscaling (adding/removing instances) with vertical scaling (resizing instances) for flexible responses. 4. Scheduling Workloads to Avoid Peak Overlaps
Where possible, schedule batch jobs, backups, or intensive tasks during known low-peak periods. This smoothes out demand curves on your shared CPU fleet.
Leverage workload insights from monitoring tools. Incorporate business calendar context (holidays, weekends). Enable maintenance windows to coincide with low demand. Common Pitfalls to Avoid Assuming vCPU counts equate to fixed compute power. vCPUs often represent hyper-threaded cores; real compute capacity fluctuates. Focusing solely on average CPU utilization. This hides important spikes that cause latency or failures. Neglecting storage and egress costs. Compute costs are not the whole picture. Ignoring P95/P99 and spike duration metrics. These percentiles and temporal factors are essential to stability. Summary Checklist: Keeping Shared CPU Fleets Stable Under Variable Workloads Collect multi-week CPU utilization data with 1–5 minute granularity. Analyze P95 and P99 CPU percentiles, including spike duration. Understand shared CPU characteristics and credit mechanics for your provider's instance types. Use tools like AWS Compute Optimizer or Azure Advisor to get right-sizing recommendations. Avoid keeping perpetually small always-on instances that mask inefficiency. Implement autoscaling policies that respond to percentile-based metrics and account for spike length. Schedule batch and heavy workloads during off-peak times to reduce burst contention. Include storage and egress costs in your cost review and optimization plans. Final Thoughts
Running a stable shared CPU fleet with variable workloads is a nuanced challenge—one that demands more than just reacting to average CPU utilization or simplistic scaling rules. By taking computingforgeeks.com https://computingforgeeks.com/shared-cpu-cloud-waste-migration-guide/ a data-driven approach using percentiles, understanding provider-specific shared CPU behavior, and leveraging native tools like AWS Compute Optimizer and Azure Advisor, you can significantly reduce cloud waste and provide reliable user experience.

Next time you review your fleets, remember: don’t just ask “What’s the average CPU utilization?” Ask instead “What do the P95 and P99 look like, and how long do those spikes last?” Your environment will thank you.