The $19,400 Question: What Does Real-Time Actually Mean?

The conference room had a Grafana dashboard that looked impressive: 500k events per second, 99.9% uptime, sub-millisecond latency. Twelve engineers had spent months building a streaming architecture with 47 Kafka topics. When someone asked which of those topics actually needed real-time processing, the room went quiet.

Three. The answer was three.

The other 44 topics served dashboards that refreshed every 15 minutes, reports generated daily, and business decisions made in weekly meetings. According to a detailed analysis shared by developer Quinn Fagundes on Dev.to, 91% of the infrastructure was serving batch jobs with extra steps. The monthly bill: $22,000. The cost after right-sizing: $3,000.

The Architecture Nobody Questions

We've developed a reflex in this industry. Data comes in, and we reach for streaming infrastructure. Kafka, Kinesis, Pulsar—the specific tool doesn't matter. What matters is that we've stopped asking whether the business actually needs data processed in milliseconds, or whether we just like the idea of it.

Fagundes identified a pattern that engineering managers see constantly but rarely name: streaming architectures built not because the SLA demanded it, but because streaming looked good on the team's collective resume. Three topics genuinely required sub-second latency for fraud detection and transaction alerts. The other 44 could have been hourly cron jobs.

The dashboard that updates every 15 minutes doesn't need an event stream. The inventory sync that runs on the hour doesn't need pub-sub. The weekly summary report definitely doesn't need real-time processing. But all of them were running through Kafka, accumulating $19,400 in monthly costs for infrastructure that solved the wrong problem elegantly.

Real-Time as SLA, Not Fashion

The framework Fagundes proposes is brutally simple: if your latency requirement is under one second, you need streaming. If it's under five minutes, micro-batching works. Anything longer than that is just batch processing, and batch processing has gotten very good over the past decade.

Fraud detection blocking a transaction in 100 milliseconds? That's real-time. That requires Kafka or something like it. An analytics dashboard that business teams check between meetings? That's batch work dressed up in streaming clothes.

According to discussions in data engineering communities, batch processing remains more cost-effective in roughly 90% of use cases. Streaming demands high-performance infrastructure, continuous monitoring, state management, and the kind of operational complexity that turns into 3 AM pages. As multiple sources on DevOps architecture note, Kafka clusters don't run themselves—they require constant tuning, scaling, and rebalancing.

The question isn't whether streaming technology is powerful. It is. The question is whether the business requirement actually needs that power, or whether we've conflated technical capability with business necessity.

Resume-Driven Architecture

There's a term for this pattern: resume-driven development. It happens when architectural decisions prioritize what engineers want to learn over what the business needs to accomplish. It's the reason teams adopt Kubernetes for applications that could run on a single server, or implement microservices for monoliths that were never going to scale beyond a handful of users.

The challenge for engineering managers is that resume-driven development isn't entirely irrational. Engineers do need to grow their skills. Technologies do evolve, and teams need to stay current. But as one analysis on resume-driven architecture points out, this approach leads to designs that no longer align with maximum impact for the organization. The infrastructure becomes a monument to what the team wanted to build, not what the business needed to run.

In Fagundes's case study, the infrastructure cost breakdown was stark: $600 per month would have covered the three genuine streaming use cases. The remaining $19,400 was overhead—technical debt disguised as technical sophistication.

The Operational Reality

Cost is one measure. The operational burden is another.

Streaming architectures create ongoing toil. Every topic needs monitoring. Every partition needs balancing. Every deployment needs careful orchestration to avoid data loss. According to enterprise Kafka management best practices, managing these systems requires specialized knowledge and substantial ongoing effort. When something breaks at 2 AM, someone has to understand not just the application logic but the streaming topology, consumer lag, and partition rebalancing.

Batch jobs break differently. They either succeed or fail. The error surface is simpler. The debugging path is clearer. The operational burden doesn't scale linearly with architectural complexity because the architecture itself is simpler.

This matters for career progression in ways that aren't obvious. Junior engineers might think that managing Kafka clusters proves technical sophistication. But senior engineers and architects are judged on different criteria: did you solve the business problem efficiently? Did you create maintainable systems? Did you make your team's lives easier or harder?

The Framework Forward

The decision tree Fagundes proposes is worth internalizing. Start with the actual latency requirement—not the aspirational one, not the one that sounds impressive in architecture reviews, but the one the business SLA actually mandates.

If you need sub-second latency and high volume, streaming infrastructure makes sense. If you need decisions made in minutes, micro-batching works. If hourly or daily updates suffice, batch processing is the answer. The framework is less about the tools than about the discipline of matching technical choices to business requirements.

This discipline requires asking uncomfortable questions. Why does this dashboard need to update in real-time if stakeholders check it once a day? Why are we streaming data that gets aggregated in a batch job anyway? Why did we build this architecture, really?

Those questions feel dangerous because they expose the gap between what we built and what was needed. But they're also the questions that separate engineering maturity from engineering theater.

What This Means for Your Next Decision

The next time someone proposes a streaming architecture, ask about the SLA. Not the theoretical latency the system could achieve, but the latency the business actually requires. Ask what happens if the data arrives in five minutes instead of five milliseconds. Ask who's going to be on call when the consumer group gets stuck.

The answers will tell you whether you're solving a business problem or building a resume.

Fagundes's team cut their infrastructure costs by 85% by asking those questions honestly. More importantly, they reduced their operational complexity, simplified their incident response, and freed up engineering time to work on problems that actually mattered to the business.

That's not a story about choosing the wrong tool. It's a story about choosing tools for the right reasons. The difference between the two is $19,400 per month and a clearer path to the kind of engineering judgment that actually advances careers.