Senior Real-Time Streaming Data Engineer

Bodyarmor; Atlanta (GA); On-site; USD 152,000 - 178,000; Full time

Job summary

The Coca-Cola Company in Atlanta, GA seeks a Senior Data Engineer specializing in Real-time Streaming to shift our enterprise from batch-reliant processing to a continuous, event-driven ecosystem.

You will design standardized streaming patterns and reference implementations to enable quick deployment of real-time data streams across the organization, targeting Fabric for Spark Streaming and Real-Time Intelligence.

Qualifications

• 8+ years in data platform architecture
• 3+ years architecting & operating production-grade streaming and Kappa architectures
• 4+ years running Spark Streaming and Apache Kafka pipelines
• Deep familiarity with stream processing engines (Spark Structured Streaming, Kafka Streams, Flink)
• Strong Scala and/or PySpark experience with stateful Structured Streaming
• Advanced knowledge of Delta Lake storage and checkpointing behaviors
• Portfolio of developer-facing streaming frameworks or configurations
• Ability to articulate complex streaming topologies and real-time concepts to developers and execs

Responsibilities

• Build a reusable streaming architecture playbook for enterprise with Kappa patterns
• Architect and templatize replay/backfill mechanisms (offset reset, checkpointing, dedupe)
• Create blueprints for sub-second real-time and near-real-time SLAs
• Package complex streaming concepts into reusable templates
• Codify end-to-end idempotency with deterministic replay
• Build metadata-driven frameworks to handle upstream schema drift
• Lead ingestion design patterns across Kafka and Fabric
• Develop dashboards to track stream health like lag, throughput, DLQ routing

Skills

Streaming architecture

Kappa architecture

Executive stakeholder comms

Log-centric data design

Frameworks & templates

Tools

Spark Streaming

Apache Kafka

Delta Lake

Fabric Real-Time Intelligence

Azure Event Hubs

Databricks Delta Live

Job description

The Coca-Cola Company in Atlanta, GA seeks a Senior Data Engineer specializing in Real-time Streaming to shift our enterprise from batch-reliant processing to a continuous, event-driven ecosystem.

You will design standardized streaming patterns and reference implementations to enable quick deployment of real-time data streams across the organization, targeting Fabric for Spark Streaming and Real-Time Intelligence.