Senior Data Engineer - Streaming

Bodyarmor; Atlanta (GA); Presencial; USD 152.000 - 178.000; Jornada completa

Descripción de la vacante

The Coca-Cola Company in Atlanta, GA seeks a Senior Data Engineer specializing in Real-time Streaming to shift our enterprise from batch-reliant processing to a continuous, event-driven ecosystem.

You will design standardized streaming patterns and reference implementations to enable quick deployment of real-time data streams across the organization, targeting Fabric for Spark Streaming and Real-Time Intelligence.

Formación

• 8+ years in data platform architecture
• 3+ years architecting & operating production-grade streaming and Kappa architectures
• 4+ years running Spark Streaming and Apache Kafka pipelines
• Deep familiarity with stream processing engines (Spark Structured Streaming, Kafka Streams, Flink)
• Strong Scala and/or PySpark experience with stateful Structured Streaming
• Advanced knowledge of Delta Lake storage and checkpointing behaviors
• Portfolio of developer-facing streaming frameworks or configurations
• Ability to articulate complex streaming topologies and real-time concepts to developers and execs

Responsabilidades

• Build a reusable streaming architecture playbook for enterprise with Kappa patterns
• Architect and templatize replay/backfill mechanisms (offset reset, checkpointing, dedupe)
• Create blueprints for sub-second real-time and near-real-time SLAs
• Package complex streaming concepts into reusable templates
• Codify end-to-end idempotency with deterministic replay
• Build metadata-driven frameworks to handle upstream schema drift
• Lead ingestion design patterns across Kafka and Fabric
• Develop dashboards to track stream health like lag, throughput, DLQ routing

Conocimientos

Streaming architecture

Kappa architecture

Executive stakeholder comms

Log-centric data design

Frameworks & templates

Herramientas

Spark Streaming

Apache Kafka

Delta Lake

Fabric Real-Time Intelligence

Azure Event Hubs

Databricks Delta Live

Descripción del empleo

## Senior Data Engineer - StreamingApply: US - GA - Atlanta: Full time: Posted Today: R-144557**Job Description Summary:**We are seeking a highly specialized engineer in Real-time Streaming to evolve Coca-Cola’s enterprise data estate from batch-reliant processing to a continuous, event-driven ecosystem. In this role, your primary mission is to design, standardize, and codify streaming and **Kappa Architecture** patterns, minimizing redundant batch layers.Rather than just building isolated data pipelines, you will architect the foundational frameworks, reference implementations, and low-code configurations that enable engineering teams across the organization to instantly deploy robust, real-time data streams. The target destination for our real-time data ecosystem is Microsoft Fabric (Spark Streaming and/or Fabric Real-Time Intelligence).The ideal candidate is a champion of log-centric data design, possesses deep expertise in enterprise Apache Kafka environments.**Core Responsibilities*** Build a reusable streaming architecture playbook for the enterprise, with a strong focus on Kappa-style patterns, event-driven processing, and near-real-time analytics on Fabric.* Architect, document, and templatize standardized replay/backfill mechanisms (offset reset + checkpoint/state management + sink rebuild/dedupe strategy).* Contextualize Processing Latency by creating distinct architectural blueprints and guardrails tailored to business SLAs: sub-second real-time and near-real-time* Abstract Streaming Complexities by packaging advanced stream concepts (stateful processing, watermarking, late-arriving data remediation, and complex windowing) into reusable templates.* Codify strict design patterns for effectively-once / end-to-end idempotency with deterministic replay.* Build metadata-driven frameworks that gracefully handle upstream schema drift without interrupting or crashing 24/7 downstream streaming queries.* Act as the internal authority on streaming design patterns across Kafka/Event Hubs, designing scalable ingestion into Fabric.* Build standardized dashboards to track critical stream health metrics, including end-to-end data lag, processing throughput, backpressure, and poison-pill routing via Dead Letter Queues (DLQ).**Required Qualifications & Experience*** 8+ years of experience in data platform architecture, with a dedicated minimum of 3+ years architecting and operating production-grade Streaming and Kappa Architectures.* 4+ years of experience running Spark Streaming and Apache Kafka pipelines. Must understand topic partitioning strategies, consumer group mechanics, schema registries (Avro/JSON), and performance tuning for high-throughput streams.* Deep familiarity with stream processing engines such as Spark Structured Streaming, Kafka Streams, or Apache Flink.* Strong **Scala and/or PySpark** experience with stateful Structured Streaming.* Advanced knowledge of the internal mechanics of Delta Lake storage, specifically regarding append-only streaming sinks and checkpointing behaviors.* The \"Framework\" Mindset: A strong portfolio demonstrating the creation of developer-facing frameworks, custom connectors, or configuration-driven streaming engines that abstract infrastructure.* Ability to articulate complex streaming topologies and real-time concepts to both application developers and executive stakeholders.**Preferred Qualifications*** Confluent Certified Developer or Kafka Administrator certification.* Proven experience modernizing Lambda architectures toward Kappa-first designs.* Hands-on experience utilizing Fabric Real-Time Intelligence, including Fabric Eventstreams and KQL Databases. *(Equivalent deep experience with Azure Event Hubs, Stream Analytics, or Databricks Delta Live).*The Coca-Cola Company will not offer sponsorship for employment status (including, but not limited to, H1-B visa status and other employment-based nonimmigrant visas) for this position. Accordingly, all applicants must be currently authorized to work in the United States on a full-time basis and must not require The Coca-Cola Company's sponsorship to continue to work legally in the United States.**Skills:****Pay Range:**United States: 152,000 - 178,300 USD*Base pay offered may vary depending on geography, job-related knowledge, skills, and experience. A full range of medical, financial, and/or other benefits, dependent on the position, is offered.***Annual Incentive Reference Value Percentage:**15*Annual Incentive reference value is a market-based competitive value for your role. It falls in the middle of the range for your role, indicating performance at target.***Location(s):**United States of America**City/Cities:**Atlanta**Travel Required:**00% - 25%**Relocation Provided:**No**Job Posting End Date:**November 6, 2026**Our Purpose and Growth Culture:**We are taking deliberate action to nurture an inclusive culture that is grounded in our company purpose, to refresh the world and make a difference. We act with a growth mindset, take an expansive approach to what’s possible and believe in continuous learning to improve our business and ourselves. We focus on four key behaviors – curious, empowered, inclusive and agile – and value how we work as much as what we achieve. We believe that our culture is one of the reasons our company continues to thrive after 130+ years. Visit Our Purpose and Vision to learn more about these behaviors and how you can bring them to life in your next role at Coca-Cola.We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, national origin, religion, sexual orientation, gender identity and/or expression, status as a veteran, and basis of disability or any other federal, state or local protected class. When we collect your personal information as part of a job application or offer of employment, we do so in accordance with industry standards and best practices and in compliance with applicable privacy laws.