We Lost 4 Hours to Ray Serve Misconfiguration (Here’s What We Changed)

Our Ray Serve deployment melted at 2 AM. These three autoscaling parameters fixed it, and we haven’t had a scaling incident since.

It was 2:17 AM on a Thursday when my phone started buzzing. Our recommendation API had gone from handling a comfortable 2,000 requests per second to a complete meltdown. The dashboard? A flatline of 502 errors. Slack was blowing up. And somewhere in our Ray Serve deployment, replicas were spinning up far too slowly to handle a viral product feature that nobody had warned the ML platform team about.

That incident cost us four hours of partial downtime and taught me more about Ray Serve autoscaling configuration than six months of reading documentation ever could. Every hard-won lesson from that night lives in this guide, along with everything we learned in the subsequent months, scaling our inference platform from 100 to 10,000 RPS.

You’re probably here because you already know Ray Serve is a powerful framework for deploying machine learning models in production with Python. But here’s what you might not realize: its autoscaling behavior can be surprisingly counterintuitive. The default configurations? They’re almost never right for production workloads.

What I’m sharing here are the exact configurations that transformed our flaky deployment into a system that now handles 10x traffic spikes without breaking a sweat. We’ll cover the internals that matter, the parameters worth tuning first, and the monitoring queries that’ll let you sleep through the night.

Ray Serve Autoscaling Internals: What the Docs Don’t Tell You

The Ray Serve autoscaler operates as a control loop that periodically evaluates scaling decisions. It looks at the current number of pending requests across all replicas, divides by your target requests per replica, and decides whether to scale up or down.

Simple enough, right? Here’s what tripped us up.

The controller doesn’t react instantly. It uses exponential smoothing to prevent thrashing, which means sudden traffic spikes get dampened before triggering scale-up decisions. During our incident, traffic increased 8x in about 90 seconds. The smoothing algorithm interpreted this as “probably temporary” and hesitated. By the time it committed to scaling, our existing replicas were drowning.

Another gotcha worth mentioning: replica startup time isn’t factored into the autoscaling decision. When your model takes 45 seconds to load (ours did, since we were loading a 4GB PyTorch model from S3), the controller will happily request new replicas while your users experience timeouts. It has no concept of “replicas that are starting but not ready yet” beyond a basic pending count.

So what’s the first step in configuring autoscaling in Ray Serve? Accepting this reality. Defaults assume your models are small and your traffic is predictable. Production ML inference rarely works that way.

The Three Critical Parameters Explained with Real Metrics

Here’s what actually moved the needle for us, with the production data to back it up.

target_num_ongoing_requests

This parameter controls how many concurrent requests each replica should handle before the controller considers scaling up. The default is 1, which sounds conservative but actually causes problems.

# What we started with (bad)
@serve.deployment(
    autoscaling_config={
        "target_num_ongoing_requests": 1,
    }
)

With the target set to 1, any queuing triggers scale up. But here’s the thing: model inference has variance. A batch that takes 50ms instead of 30ms temporarily queues the next request, triggering unnecessary scaling. We saw our replica count oscillate between 4 and 12 constantly. It was chaos.

# What actually works for our latency-sensitive API
@serve.deployment(
    autoscaling_config={
        "target_num_ongoing_requests": 3,
    }
)

Setting it to 3 gave each replica headroom for normal variance while still scaling when genuinely overloaded. Our p99 latency actually improved because we stopped wasting resources on constant scale-up/scale-down cycles. Who would’ve thought?

smoothing_factor

This parameter controls how much the controller trusts recent observations versus historical averages. Higher values mean faster reactions.

During our outage, we were running the default of 0.5. Bumping it to 0.8 cut our reaction time from ~15 seconds to ~5 seconds for genuine traffic spikes.

autoscaling_config={
    "smoothing_factor": 0.8,  # React faster to changes
}

But here’s the tradeoff you need to know about: higher smoothing factors also mean more sensitivity to noise. Got genuinely bursty traffic from flash sales, marketing emails, or social media mentions? You want this high. Steady traffic with occasional one-off spikes? Keep it lower.

downscale_delay_s

This parameter determines how long the controller waits after the load decreases before removing replicas. The default is quite conservative, which might be excessive for most workloads.

We dropped it to 120 seconds for our cost-optimized deployments and kept it at 300 seconds for latency-critical paths.

autoscaling_config={
    "downscale_delay_s": 120,  # 2 minutes instead of the default
}

Our reasoning: if traffic has been low for 2 minutes, it’s probably actually low. But don’t go below 60 seconds unless you’ve got very predictable traffic. The cost of scaling back up (replica startup time, cold cache effects) usually exceeds the savings from aggressive downscaling. I learned that one the hard way.

Configuration Templates for Different Traffic Patterns

Below are copy-paste configs for the three most common scenarios I encounter. We’ve validated all of these in production for scaling real-time ML inference workloads.

Bursty Traffic Configuration

@serve.deployment(
    ray_actor_options={"num_cpus": 2},
    autoscaling_config={
        "min_replicas": 3,
        "max_replicas": 50,
        "initial_replicas": 5,
        "target_num_ongoing_requests": 2,
        "smoothing_factor": 0.85,
        "upscale_delay_s": 3,
        "downscale_delay_s": 180,
    },
)
class BurstyTrafficModel:
    # Your model code here
    pass

Keep a minimum of replicas higher than you think you need. The goal? Handling the first wave of a spike while new replicas come online.

Steady Load Configuration

@serve.deployment(
    ray_actor_options={"num_cpus": 2},
    autoscaling_config={
        "min_replicas": 2,
        "max_replicas": 20,
        "initial_replicas": 4,
        "target_num_ongoing_requests": 5,
        "smoothing_factor": 0.5,
        "upscale_delay_s": 10,
        "downscale_delay_s": 300,
    },
)
class SteadyLoadModel:
    pass

Higher target requests per replica and slower reactions. You’re optimizing for efficiency here, not reaction speed.

Cost-Optimized Configuration

@serve.deployment(
    ray_actor_options={"num_cpus": 1},
    autoscaling_config={
        "min_replicas": 1,
        "max_replicas": 10,
        "initial_replicas": 1,
        "target_num_ongoing_requests": 8,
        "smoothing_factor": 0.4,
        "upscale_delay_s": 15,
        "downscale_delay_s": 90,
    },
)
class CostOptimizedModel:
    pass

This configuration accepts higher latency variance in exchange for lower compute costs. Only use it for non-user-facing workloads or internal tools. Seriously.

Ray Serve vs. TensorFlow Serving: Autoscaling Response Time Observations

I ran a controlled comparison because I kept getting asked this question. Honestly, it comes down to one main factor for autoscaling: how quickly can the system respond to load changes?

Test setup: Same PyTorch model exported to both TorchServe (for TensorFlow Serving comparison) and Ray Serve. The traffic pattern was a 100 RPS baseline with synthetic 10x spikes lasting 60 seconds.

Autoscaling Response Time (time to reach target replica count), results from our testing:

Note: These numbers reflect our specific test environment and workload. Your results will likely vary based on model size, infrastructure, and configuration.

I’ll admit the K8s HPA comparison is a bit unfair since HPA polls metrics every 15 seconds by default. But it reflects real-world production ML patterns that many teams actually use.

Ray Serve’s native autoscaling, when properly configured, beats external orchestration because it has direct visibility into request queues. No waiting for Prometheus scrapes or metric aggregation delays.

Monitoring and Alerting: Prometheus Queries That Save You

These are the actual PromQL queries running in our alerting system. They’ve caught autoscaling issues before users noticed. Multiple times.

Replica Starvation Alert

# Alert when requests are queuing but replicas aren't scaling
(
  ray_serve_deployment_queued_queries{deployment="your_model"}
  / 
  ray_serve_deployment_replica_healthy_count{deployment="your_model"}
) > 10
and
rate(ray_serve_deployment_replica_healthy_count{deployment="your_model"}[5m]) == 0

This catches the scenario where your autoscaler is stuck. Maybe it’s hitting max replicas, maybe it’s failing to provision, maybe it’s misconfigured. Either way, you’ll want to know.

Premature Downscaling Alert

# Alert on rapid scale down followed by immediate scale up
increase(ray_serve_deployment_replica_healthy_count{deployment="your_model"}[5m]) > 0
and
increase(ray_serve_deployment_replica_healthy_count{deployment="your_model"} offset 5m)[5m] < 0

Seeing this pattern frequently? Your downscale_delay_s is too aggressive.

Latency Degradation During Scaling

# Track latency increase during autoscaling events
histogram_quantile(0.99, 
  rate(ray_serve_deployment_request_latency_ms_bucket{deployment="your_model"}[1m])
) > 500
and on() 
changes(ray_serve_deployment_replica_healthy_count{deployment="your_model"}[5m]) > 0

This tells you whether scaling events themselves are causing latency problems, which suggests cold start issues you’ll need to address.

Advanced Patterns for Production

Multi-Model Deployments

When you serve PyTorch models with Ray Serve across multiple endpoints, each model needs independent scaling configs. Don’t make the mistake of sharing one config across models with different latency characteristics.

# Composition pattern for multi-model inference
@serve.deployment(autoscaling_config=EMBEDDING_CONFIG)
class EmbeddingModel:
    pass

@serve.deployment(autoscaling_config=RANKING_CONFIG)
class RankingModel:
    pass

@serve.deployment(autoscaling_config=ORCHESTRATOR_CONFIG)
class Orchestrator:
    def __init__(self):
        self.embedding = EmbeddingModel.get_handle()
        self.ranking = RankingModel.get_handle()

Your orchestrator should have the most aggressive scaling since it’s your entry point. Downstream models can be slightly more conservative.

GPU Autoscaling Considerations

GPU replicas are expensive and slow to start. For reducing inference latency in production ML systems with GPUs, here’s what I’d recommend:

autoscaling_config={
    "min_replicas": 2,  # Always warm
    "max_replicas": 8,
    "target_num_ongoing_requests": 1,  # GPUs batch internally
    "downscale_delay_s": 600,  # Long delay, startup is expensive
}

Never set min_replicas to 0 for GPU deployments unless you’re comfortable with 60+ second cold starts. Trust me on this one.

Cross-Region Failover

For real-time prediction API architectures with high availability requirements, we run Ray Serve clusters in multiple regions behind a global load balancer. Each cluster runs its own autoscaling independently.

Here’s the trick: set consistent min_replicas across regions so failover doesn’t require scale-up. When region A handles 5,000 RPS normally, region B should maintain enough warm capacity to absorb that load immediately.

This is what I wish someone had handed me before that 2 AM incident. Use it as a starting point for implementing real-time ML inference with Ray Serve.

Before going to production:

• Measure your model’s p99 inference latency under load
• Calculate target_num_ongoing_requests as: (acceptable latency / actual latency) × 0.7
• Set smoothing_factor based on traffic predictability (0.5–0.6 for steady, 0.8+ for bursty)
• Measure replica startup time and factor it into your min_replicas calculation
• Configure downscale_delay_s to be at least 2x your typical traffic lull duration
• Set up the three monitoring alerts above

When should you consider alternatives?

Ray Serve autoscaling works well for most workloads, but consider alternatives when you need sub-second scale-up (use AWS Lambda or similar), have extremely predictable traffic (just use fixed replicas), or your models are stateless and tiny (consider serverless inference).

Everything in this Ray Serve autoscaling configuration guide took us months to learn. These configurations have handled Black Friday traffic, viral moments, and yes, the occasional 2 AM surprise. Tune them for your specific workload, monitor religiously, and you’ll build low-latency ML inference best practices into your system from the start.

And maybe keep your phone on vibrate anyway. Just in case.