Why Our Beautiful 7-Day Aggregation Feature Was Useless at Serving Time

Our fraud model dropped from 97% to 60% accuracy in production. The culprit? Features that took hours to compute but needed to serve in 50ms.

I still remember the moment our fraud detection model went from 97% accuracy in offline evaluation to catching barely 60% of fraudulent transactions in production. Nothing about the model had changed. The features had.

We’d trained on carefully aggregated behavioral signals computed over hours of batch processing. But at serving time? We had 50 milliseconds. Total. That beautiful “user_transaction_velocity_7d” feature that worked so well in training simply couldn’t be computed fast enough when a transaction hit our API.

Most ML teams eventually face this real-time feature engineering wake-up call. Honestly, it’s one I wish I’d gotten earlier in my career.

Most content on this topic glosses over the fundamental tension between what makes models accurate and what makes them servable. I’ve spent years working on experimentation platforms, watching teams stumble on this exact problem, and I’ve seen it play out at vastly different scales.

What follows is a complete walkthrough of the architecture decisions you’ll need to make when feature computation latency becomes a hard constraint. We’ll cover the taxonomy of feature latency, streaming pipeline patterns, the heated feature store versus real-time computation debate, and three battle-tested architectures I’ve seen work across different scale requirements.

Whether you’re serving 100 requests per second or 100,000, the core principles remain consistent. But the implementations? They differ wildly.

The Feature Latency Taxonomy: Classifying Your Features Before Building Anything

Before you write a single line of streaming code, you need to classify every feature in your model by two dimensions: computation cost and freshness requirements.

Static Features (Pre-computed, Near-Zero Serving Latency)

These are your user demographic attributes, product catalog information, and geographic lookups. They change infrequently and can be cached aggressively. Serving cost: a single key-value lookup, typically under 1ms.

Sliding Window Aggregations (Medium Complexity)

Think “count of user logins in last 7 days” or “average order value this month.” These require maintaining state over time windows. You’ve got two options: pre-compute and store, or compute incrementally as events arrive.

Real-Time Session Features (High Freshness, Moderate Cost)

Current session duration. Items viewed in this visit. These need to reflect events from seconds or milliseconds ago. Caching helps, but staleness kills utility.

Request-Time Computations (Maximum Freshness, Highest Cost)

These features depend on the specific inference request itself. Distance between the user location and the nearest store. Similarity between the search query and the product description. No pre-computation is possible.

Working on experimentation platforms taught me that most teams overestimate how many features actually need request-time computation. Auditing feature sets typically reveal that 60–70% of features can be pushed into sliding window aggregations with creative windowing strategies.

Your first exercise: take your current feature set and plot each feature on a 2×2 matrix of computation cost versus staleness tolerance. Features landing in the high-cost, low-staleness quadrant? Those are your architectural bottlenecks.

Streaming Feature Pipelines for Machine Learning: Kafka, Flink, and Spark Patterns

If you’re trying to build streaming feature pipelines for machine learning in 2025 and you haven’t committed to either Kafka/Flink or Kafka/Spark Structured Streaming as your backbone, you’re making your life unnecessarily difficult.

Kafka as the Foundation

Kafka sits at the center of every streaming pipeline I’ve built or inherited. Events flow in, and you need to transform them into feature updates. So what does the transformation layer look like?

Flink for Sub-Second Feature Updates

Flink excels when you need true low-latency feature computation techniques. Its event-time processing semantics handle out-of-order events gracefully, which matters enormously for windowed aggregations.

A pattern I’ve seen work well:

# Pseudocode for Flink session window aggregation
stream = env.add_source(kafka_consumer)
    .key_by(lambda event: event.user_id)
    .window(SessionWindows.with_gap(Time.minutes(30)))
    .aggregate(
        SessionFeatureAggregator(),
        output_type=SessionFeatures
    )
    .add_sink(feature_store_sink)

Flink lets you maintain complex state (like session windows with arbitrary gaps) while still pushing updates to your feature store within milliseconds of the window closing. That’s the real power here.

Spark Structured Streaming for Throughput

Slightly relaxed latency requirements (1–5 seconds acceptable), but massive throughput? Spark Structured Streaming often makes more sense. A mature ecosystem, a larger talent pool, and clean integration with existing Spark batch jobs simplify your architecture.

# Spark Structured Streaming equivalent
features = (spark
    .readStream
    .format("kafka")
    .load()
    .groupBy(
        window("timestamp", "1 hour", "5 minutes"),
        "user_id"
    )
    .agg(count("*").alias("event_count"))
    .writeStream
    .format("feature_store_connector")
    .outputMode("update")
    .start())

A Hybrid Reality

Most production systems I’ve worked on end up with both. Flink handles the truly latency-sensitive features (fraud signals, real-time bidding features), while Spark handles the bulk of near-real-time aggregations. Your feature store becomes the unification layer.

Feature Store vs. Real-Time Computation: Making the Right Call

I once watched two senior engineers nearly come to blows in a design review over whether to use Tecton or build custom Flink pipelines. One insisted the abstraction was worth the cost; the other called it “expensive training wheels.” They were both right, depending on what you optimized for. That’s why this debate generates surprisingly heated arguments in ML infrastructure circles.

After evaluating these systems across multiple organizations, I’ve landed on some guidelines.

Feature Stores Make Sense When…

Feature stores (Feast, Tecton, Featureform, and others) excel when:

• You have training-serving skew problems (and you almost certainly do)
• Multiple models share the same features
• Point-in-time correctness matters for training
• You need feature versioning and lineage

Computing at Request Time Makes Sense When…

Skip the feature store and compute directly when:

• Features depend entirely on request-time inputs
• Freshness requirements are measured in milliseconds
• Computation is cheap enough to fit in your latency budget
• Features are specific to one model and unlikely to be reused

A Real Comparison: Feast vs. Tecton vs. Featureform in 2025

Having worked with all three, here’s my honest assessment:

Feast remains the best open-source option. It’s flexible, runs anywhere, and has an active community. But you’re building significant infrastructure yourself. If you have a strong platform team, this is often the right choice.

Tecton is the managed solution that handles the hard parts: streaming transformations, backfills, and monitoring. A real price tag, but real engineering time saved too. Best for teams that want to move fast and have a budget.

Featureform positions itself as the “virtual feature store” that sits on top of your existing infrastructure. Works well if you’re already invested in Spark, Snowflake, or Databricks and don’t want to migrate data. Occasionally, the abstraction leaks, though.

For most teams building their first real-time feature engineering system, I suggest starting with Feast to understand the concepts, then evaluating managed alternatives once you hit scale pain.

Point-in-Time Correctness: The Silent Model Killer

Point-in-time correctness in feature engineering, explained in one sentence: when you train your model, you should only see the features that would have been available at prediction time.

Sounds obvious. And yet it’s shockingly easy to violate.

Consider a scenario that’s bitten teams repeatedly: you compute a feature like “user_booking_count_30d” using the current value from your data warehouse, join it to historical labels, and train a model. Looks great. In production, it performs terribly.

Why? Because the data warehouse values reflected the current state, not the state at the time of each historical prediction. You were essentially training with future information leakage.

Guaranteeing Point-in-Time Correctness

Two components solve this problem:

Temporal feature tables. Every feature value must be stored with an effective timestamp, not just the “current” value. Creating training data means joining on the timestamp that was active at label time.

Snapshot isolation. Your feature retrieval for training must be “as-of” a specific timestamp. Modern feature stores handle this, but if you’re rolling your own, you need explicit logic.

-- Point-in-time correct feature retrieval
SELECT l.user_id, l.label, l.event_timestamp, f.feature_value
FROM labels l
LEFT JOIN features f ON l.user_id = f.user_id
  AND f.feature_timestamp <= l.event_timestamp
  AND f.feature_timestamp > l.event_timestamp - INTERVAL '30 days'
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY l.user_id, l.event_timestamp 
  ORDER BY f.feature_timestamp DESC
) = 1

Verbose, but correct. You only see the most recent feature value that existed before the label event.

Architecture Patterns That Actually Work at Different Scales

Let me describe three architectures for feature engineering for online ML inference that I’ve seen succeed in production. Each optimizes for different constraints.

Pattern 1: Pre-Compute Everything (Simple, Higher Latency Tolerance)

Choose this if: You’re a team with lower throughput requirements and more relaxed latency budgets, valuing simplicity over real-time freshness.

Architecture:

• Batch jobs compute features daily or hourly
• Features land in Redis or DynamoDB
• Serving layer does simple key lookups
• Request-time features computed in application code

Most teams should start here. Honestly, most guides on this topic overcomplicate things for teams that don’t actually need sub-50ms latency.

Pattern 2: Streaming Plus Store Hybrid (Balanced)

Suited to teams that have moderate to high throughput needs and latency budgets in the tens to low hundreds of milliseconds, where feature freshness measured in seconds provides meaningful value.

Architecture:

• Kafka ingests events
• Flink or Spark computes sliding window aggregations
• The feature store serves pre-computed features
• Thin request-time computation layer for personalization features

Most mature ML teams land here. You get feature freshness measured in seconds (not hours), reasonable latency, and manageable infrastructure complexity.

Pattern 3: Real-Time Computation Engine (Maximum Freshness)

Works when: You have very high throughput scenarios with extremely tight latency requirements, and each millisecond of delay has a measurable business impact.

Architecture:

• In-memory feature computation engines (custom or vendor)
• Heavy caching at multiple layers
• Approximation algorithms for expensive aggregations
• Aggressive feature selection to minimize computation

This is how you reduce feature computation latency in production when milliseconds matter for revenue. Think ad bidding, high-frequency trading, and real-time pricing. You’ll need dedicated infrastructure engineers and significant investment.

Practical Next Steps

My recommended path for adopting real-time feature engineering best practices:

Step 1: Audit and Classify (Week 1)

Create a spreadsheet with every feature in your production models. For each document:

• Current computation method
• Staleness tolerance (minutes? hours? days?)
• Computation cost estimate
• Freshness requirement (how recent must it be?)

Step 2: Identify the Bottlenecks (Week 2)

Which features are impossible to serve within your latency budget? Can they be approximated? Pre-computed? Or do they need streaming infrastructure?

Step 3: Choose Your Pattern (Week 3)

Based on your scale and latency requirements, pick one of the three architecture patterns. Don’t over-engineer. Pattern 1 works for more use cases than people admit.

Step 4: Start with One Feature Pipeline (Month 1)

Build the streaming pipeline for your single most valuable real-time feature. Get it to production. Learn what breaks.

Step 5: Evaluate Feature Stores (Month 2–3)

Once you understand your requirements through direct experience, evaluate whether a managed feature store makes sense. Sometimes the answer is no.

That 50ms budget changes everything because it forces you to make architectural decisions you’d otherwise defer. But constraints create clarity. And clarity, in my experience, leads to systems that actually work in production.

Start measuring. Then start building. Because the models you train are only as good as the features you can actually serve.

Disclaimer: This content is for educational purposes only and does not constitute financial advice.