Senior Data Engineer - Streaming

0801 The Coca-Cola Company; Atlanta (GA); On-site; USD 152,000 - 178,000; Full time

Job summary

The Coca-Cola Company in Atlanta is seeking a highly specialized Real-time Streaming Engineer to lead the move from batch processing to a continuous, event-driven data ecosystem. You will design and codify streaming patterns, build reusable frameworks, and enable teams across the enterprise to deploy real-time data streams on Microsoft Fabric.

The role emphasizes log-centric data design, Kafka expertise, and the creation of templates, tooling, and dashboards to monitor end-to-end latency,

Qualifications

• 8+ years of experience in data platform architecture, with a dedicated minimum of 3+ years architecting and operating production-grade Streaming and Kappa Architectures.
• 4+ years of experience running Spark Streaming and Apache Kafka pipelines.
• Must understand topic partitioning strategies, consumer group mechanics, schema registries (Avro/JSON), and performance tuning for high-throughput streams.
• Deep familiarity with stream processing engines such as Spark Structured Streaming, Kafka Streams, or Apache Flink.
• StrongScala and/or PySparkexperience with stateful Structured Streaming.
• Advanced knowledge of the internal mechanics of Delta Lake storage, specifically regarding append-only streaming sinks and checkpointing behaviors.
• The "Framework" Mindset: A strong portfolio demonstrating the creation of developer-facing frameworks, custom connectors, or configuration-driven streaming engines that abstract infrastructure.
• Ability to articulate complex streaming topologies and real-time concepts to both application developers and executive stakeholders.

Responsibilities

• Build a reusable streaming architecture playbook for the enterprise, with a strong focus on Kappa-style patterns, event-driven processing, and near-real-time analytics on Fabric.
• Architect, document, and templatize standardized replay/backfill mechanisms (offset reset + checkpoint/state management + sink rebuild/dedupe strategy).
• Contextualize Processing Latency by creating distinct architectural blueprints and guardrails tailored to business SLAs: sub-second real-time and near-real-time.
• Abstract Streaming Complexities by packaging advanced stream concepts (stateful processing, watermarking, late-arriving data remediation, and complex windowing) into reusable templates.
• Codify strict design patterns for effectively-once / end-to-end idempotency with deterministic replay.
• Build metadata-driven frameworks that gracefully handle upstream schema drift without interrupting or crashing 24/7 downstream streaming queries.
• Act as the internal authority on streaming design patterns across Kafka/Event Hubs, designing scalable ingestion into Fabric.
• Build standardized dashboards to track critical stream health metrics, including end-to-end data lag, processing throughput, backpressure, and poison-pill routing via Dead Letter Queues (DLQ).

Skills

Data platform architecture

Kafka & streaming

Spark Streaming

Scala / PySpark

Delta Lake

Frameworks & templates

Stakeholder communication

Tools

Fabric Real-Time Intelligence

Azure Event Hubs

Databricks Delta Live

Job description

Job Description Summary

We are seeking a highly specialized engineer in Real-time Streaming to evolve Coca-Cola’s enterprise data estate from batch-reliant processing to a continuous, event-driven ecosystem. In this role, your primary mission is to design, standardize, and codify streaming and Kappa Architecture patterns, minimizing redundant batch layers. Rather than just building isolated data pipelines, you will architect the foundational frameworks, reference implementations, and low-code configurations that enable engineering teams across the organization to instantly deploy robust, real-time data streams. The target destination for our real-time data ecosystem is Microsoft Fabric (Spark Streaming and/or Fabric Real-Time Intelligence). The ideal candidate is a champion of log-centric data design, possesses deep expertise in enterprise Apache Kafka environments.

Core Responsibilities

• Build a reusable streaming architecture playbook for the enterprise, with a strong focus on Kappa-style patterns, event-driven processing, and near-real-time analytics on Fabric.
• Architect, document, and templatize standardized replay/backfill mechanisms (offset reset + checkpoint/state management + sink rebuild/dedupe strategy).
• Contextualize Processing Latency by creating distinct architectural blueprints and guardrails tailored to business SLAs: sub-second real-time and near-real-time.
• Abstract Streaming Complexities by packaging advanced stream concepts (stateful processing, watermarking, late-arriving data remediation, and complex windowing) into reusable templates.
• Codify strict design patterns for effectively-once / end-to-end idempotency with deterministic replay.
• Build metadata-driven frameworks that gracefully handle upstream schema drift without interrupting or crashing 24/7 downstream streaming queries.
• Act as the internal authority on streaming design patterns across Kafka/Event Hubs, designing scalable ingestion into Fabric.
• Build standardized dashboards to track critical stream health metrics, including end-to-end data lag, processing throughput, backpressure, and poison-pill routing via Dead Letter Queues (DLQ).

Required Qualifications & Experience

• 8+ years of experience in data platform architecture, with a dedicated minimum of 3+ years architecting and operating production-grade Streaming and Kappa Architectures.
• 4+ years of experience running Spark Streaming and Apache Kafka pipelines.
• Must understand topic partitioning strategies, consumer group mechanics, schema registries (Avro/JSON), and performance tuning for high-throughput streams.
• Deep familiarity with stream processing engines such as Spark Structured Streaming, Kafka Streams, or Apache Flink.
• StrongScala and/or PySparkexperience with stateful Structured Streaming.
• Advanced knowledge of the internal mechanics of Delta Lake storage, specifically regarding append-only streaming sinks and checkpointing behaviors.
• The "Framework" Mindset: A strong portfolio demonstrating the creation of developer-facing frameworks, custom connectors, or configuration-driven streaming engines that abstract infrastructure.
• Ability to articulate complex streaming topologies and real-time concepts to both application developers and executive stakeholders.

Preferred Qualifications

• Confluent Certified Developer or Kafka Administrator certification.
• Proven experience modernizing Lambda architectures toward Kappa-first designs.
• Hands-on experience utilizing Fabric Real-Time Intelligence, including Fabric Eventstreams and KQL Databases. (Equivalent deep experience with Azure Event Hubs, Stream Analytics, or Databricks Delta Live).

Skills

Pay Range: United States: 152,000 - 178,300 USD

Base pay offered may vary depending on geography, job-related knowledge, skills, and experience.

A full range of medical, financial, and/or other benefits, dependent on the position, is offered.

Annual Incentive Reference Value Percentage: 15

Annual Incentive reference value is a market-based competitive value for your role. It falls in the middle of the range for your role, indicating performance at target.

Location(s): United States of America

City/Cities: Atlanta

Travel Required: 00% - 25%

Relocation Provided: No

Job Posting End Date: November 6, 2026

Our Purpose and Growth Culture

We are taking deliberate action to nurture an inclusive culture that is grounded in our company purpose, to refresh the world and make a difference. We act with a growth mindset, take an expansive approach to what’s possible and believe in continuous learning to improve our business and ourselves. We focus on four key behaviors – curious, empowered, inclusive and agile – and value how we work as much as what we achieve. We believe that our culture is one of the reasons our company continues to thrive after 130+ years. Visit Our Purpose and Vision to learn more about these behaviors and how you can bring them to life in your next role at Coca-Cola.

We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, national origin, religion, sexual orientation, gender identity and/or expression, status as a veteran, and basis of disability or any other federal, state or local protected class.

When we collect your personal information as part of a job application or offer of employment, we do so in accordance with industry standards and best practices and in compliance with applicable privacy laws.

Additional Information

The Coca-Cola Company is a global community of passionate employees who are refreshing the world and making a difference every day. Innovation has been at the heart of our story since 1886. It goes beyond new flavors and brands. Learn more about our system and how we’re making an impact. As a global organization, we seek opportunities to embrace and champion progress. Our careers span across functions and are globally connected to give you an unmatched career experience. We’re glad you’ve chosen to explore your next career opportunity with The Coca-Cola Company. Learn more about our hiring process and pick up tips on how to showcase your experience and skills so you can stand out and thrive. The Coca‑Cola global business system is composed of The Coca‑Cola Company (TCCC), its subsidiaries and 225 bottling partners around the world. Search jobs throughout our system.