Real-Time Streaming Architect: Kafka/Kappa on Fabric

0801 The Coca-Cola Company; Atlanta (GA); On-site; USD 152,000 - 178,000; Full time

Job summary

The Coca-Cola Company in Atlanta is seeking a highly specialized Real-time Streaming Engineer to lead the move from batch processing to a continuous, event-driven data ecosystem. You will design and codify streaming patterns, build reusable frameworks, and enable teams across the enterprise to deploy real-time data streams on Microsoft Fabric.

The role emphasizes log-centric data design, Kafka expertise, and the creation of templates, tooling, and dashboards to monitor end-to-end latency,

Qualifications

• 8+ years of experience in data platform architecture, with a dedicated minimum of 3+ years architecting and operating production-grade Streaming and Kappa Architectures.
• 4+ years of experience running Spark Streaming and Apache Kafka pipelines.
• Must understand topic partitioning strategies, consumer group mechanics, schema registries (Avro/JSON), and performance tuning for high-throughput streams.
• Deep familiarity with stream processing engines such as Spark Structured Streaming, Kafka Streams, or Apache Flink.
• StrongScala and/or PySparkexperience with stateful Structured Streaming.
• Advanced knowledge of the internal mechanics of Delta Lake storage, specifically regarding append-only streaming sinks and checkpointing behaviors.
• The "Framework" Mindset: A strong portfolio demonstrating the creation of developer-facing frameworks, custom connectors, or configuration-driven streaming engines that abstract infrastructure.
• Ability to articulate complex streaming topologies and real-time concepts to both application developers and executive stakeholders.

Responsibilities

• Build a reusable streaming architecture playbook for the enterprise, with a strong focus on Kappa-style patterns, event-driven processing, and near-real-time analytics on Fabric.
• Architect, document, and templatize standardized replay/backfill mechanisms (offset reset + checkpoint/state management + sink rebuild/dedupe strategy).
• Contextualize Processing Latency by creating distinct architectural blueprints and guardrails tailored to business SLAs: sub-second real-time and near-real-time.
• Abstract Streaming Complexities by packaging advanced stream concepts (stateful processing, watermarking, late-arriving data remediation, and complex windowing) into reusable templates.
• Codify strict design patterns for effectively-once / end-to-end idempotency with deterministic replay.
• Build metadata-driven frameworks that gracefully handle upstream schema drift without interrupting or crashing 24/7 downstream streaming queries.
• Act as the internal authority on streaming design patterns across Kafka/Event Hubs, designing scalable ingestion into Fabric.
• Build standardized dashboards to track critical stream health metrics, including end-to-end data lag, processing throughput, backpressure, and poison-pill routing via Dead Letter Queues (DLQ).

Skills

Data platform architecture

Kafka & streaming

Spark Streaming

Scala / PySpark

Delta Lake

Frameworks & templates

Stakeholder communication

Tools

Fabric Real-Time Intelligence

Azure Event Hubs

Databricks Delta Live

Job description

The Coca-Cola Company in Atlanta is seeking a highly specialized Real-time Streaming Engineer to lead the move from batch processing to a continuous, event-driven data ecosystem. You will design and codify streaming patterns, build reusable frameworks, and enable teams across the enterprise to deploy real-time data streams on Microsoft Fabric.

The role emphasizes log-centric data design, Kafka expertise, and the creation of templates, tooling, and dashboards to monitor end-to-end latency,