Senior Data Engineer - Streaming

The Coca-Cola Company; Atlanta (GA); On-site; USD 150,000 - 230,000; Full time

Job summary

The Coca-Cola Company seeks a highly specialized engineer in Real-time Streaming to evolve its enterprise data estate from batch processing to a continuous, event-driven ecosystem. You will design, standardize, and codify streaming patterns and low-code configurations to enable rapid, reliable real-time data delivery across the organization, targeting Microsoft Fabric for streaming workloads.

Ideal candidates bring deep Kafka expertise and a proven track record in building scalable streaming

Qualifications

• 8+ years in data platform architecture with streaming experience.
• 4+ years running Spark Streaming and Kafka pipelines with high throughput.
• Deep familiarity with Spark Structured Streaming, Kafka Streams, or Flink.
• Strong Scala and/or PySpark experience with stateful streaming.

Responsibilities

• Build a reusable streaming architecture playbook with Kappa patterns and near-real-time analytics on Fabric.
• Architect and template standardized replay/backfill mechanisms and checkpointing strategies.
• Define guardrails for sub-second latency and business SLA alignment.
• Abstract streaming complexities into reusable templates (stateful processing, watermarking, late data).
• Codify end-to-end idempotency with deterministic replay and dashboards for stream health.

Skills

Kafka

Spark Streaming

Scala

PySpark

Delta Lake

Fabric Real-Time Intelligence

Tools

Apache Spark

Apache Kafka

Delta Lake

Azure Event Hubs

Databricks Delta Live

Job description

We are seeking a highly specialized engineer in Real-time Streaming to evolve Coca-Cola’s enterprise data estate from batch-reliant processing to a continuous, event-driven ecosystem. In this role, your primary mission is to design, standardize, and codify streaming and Kappa Architecture patterns, minimizing redundant batch layers.

Rather than just building isolated data pipelines, you will architect the foundational frameworks, reference implementations, and low-code configurations that enable engineering teams across the organization to instantly deploy robust, real-time data streams. The target destination for our real-time data ecosystem is Microsoft Fabric (Spark Streaming and/or Fabric Real-Time Intelligence).

The ideal candidate is a champion of log-centric data design, possesses deep expertise in enterprise Apache Kafka environments.

Core Responsibilities

• Build a reusable streaming architecture playbook for the enterprise, with a strong focus on Kappa-style patterns, event-driven processing, and near-real-time analytics on Fabric.

• Architect, document, andtemplatizestandardized replay/backfill mechanisms (offset reset + checkpoint/state management + sink rebuild/dedupe strategy).

• Contextualize Processing Latencyby creatingdistinct architectural blueprints and guardrails tailored to business SLAs:sub-second real-time and near-real-time

• Abstract Streaming Complexitiesby packagingadvanced stream concepts(stateful processing, watermarking, late-arriving data remediation, and complex windowing)intoreusabletemplates.

• Codify strict design patternsforeffectively-once /end-to-end idempotency with deterministic replay.

• Build metadata-driven frameworks that gracefully handle upstream schema drift without interrupting or crashing 24/7 downstream streaming queries.

• Act as the internal authority on streaming design patterns across Kafka/Event Hubs, designingscalableingestion into Fabric.

• Build standardized dashboards to track critical stream health metrics, including end-to-end data lag, processing throughput, backpressure, and poison-pill routing via Dead Letter Queues (DLQ).

Required Qualifications & Experience

• 8+ years of experience in data platform architecture, with a dedicated minimum of 3+yearsarchitecting and operating production-grade Streaming and Kappa Architectures.

• 4+ years ofexperiencerunningSpark Streaming andApache Kafkapipelines. Must understand topic partitioning strategies, consumer group mechanics, schema registries (Avro/JSON), and performance tuning for high-throughput streams.

• Deep familiarity with stream processing engines such as Spark Structured Streaming, Kafka Streams, or Apache Flink.

• Strong Scala and/or PySpark experience with stateful Structured Streaming.

• Advanced knowledge of the internal mechanics of Delta Lake storage, specifically regarding append-only streaming sinks and checkpointing behaviors.

• The "Framework" Mindset: A strong portfoliodemonstratingthe creation of developer-facing frameworks, custom connectors, or configuration-driven streaming engines that abstract infrastructure.

• Abilityto articulate complex streaming topologies and real-time concepts to both application developers and executive stakeholders.

Preferred Qualifications

• Confluent Certified Developer or Kafka Administrator certification.

• Proven experience modernizing Lambda architectures toward Kappa-first designs.

• Hands-on experienceutilizingFabric Real-Time Intelligence, including FabricEventstreamsand KQL Databases. (Equivalent deep experience with Azure Event Hubs, Stream Analytics, or Databricks Delta Live).

We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, national origin, religion, sexual orientation, gender identity and/or expression, status as a veteran, and basis of disability or any other federal, state or local protected class.