Lead Data Engineer - Real-Time Pipelines & AI Systems

Empower Retirement; United States; Hybrid; USD 140,000 - 210,000; Full time

Job summary

Empower Retirement is seeking a Lead Data Engineer to guide the design, development, and optimization of scalable data pipelines and real-time ingestion platforms across enterprise data ecosystems. You will mentor engineers, collaborate with architects, and drive automation, governance, and performance improvements using Kafka, Spark, Flink, and cloud technologies.

This role emphasizes resilience, security, and data quality, with a broad remit across streaming, lakehouse formats, and schema

Qualifications

• Bachelor's degree in computer science or related field.
• 6-8+ years of data engineering on enterprise-scale platforms.
• Hands-on with Kafka, Confluent Cloud, AWS Kinesis, or Pulsar.
• Proficient in Python, PySpark, SQL.
• Experience with Docker, Kubernetes, Terraform, CI/CD.
• Strong knowledge of CDC and schema evolution (Schema Registry).
• Experience with lakehouse formats (Iceberg/Delta Lake) and Snowflake/BigQuery.
• Excellent communication and Agile experience.

Responsibilities

• Lead design of data pipelines, ingestion frameworks, and real-time streaming.
• Build/optimize Kafka producers/consumers for sub-second latency.
• Configure CDC solutions across relational/NoSQL to Kafka topics.
• Develop stream processing using Flink or Spark.
• Implement schema evolution with Schema Registry.
• Integrate automated code generation and LLM orchestration.
• Implement pipeline monitoring with Prometheus, Grafana, Datadog.
• Mentor engineers and drive architectural decisions.

Skills

Apache Kafka

Confluent Cloud

AWS Kinesis

Apache Pulsar

Python

PySpark

SQL

Java

Scala

Go

Flink

Kafka Streams

Spark Streaming

Iceberg

Delta Lake

Snowflake

BigQuery

Docker

Kubernetes

Terraform

CI/CD

Schema Registry

CDC

ETL/ELT

Data Warehousing

Data Modeling

Agile

Leadership

Mentoring

Education

Bachelor's degree in computer science or related field

Tools

Docker

Kubernetes

Terraform

CI/CD

Schema Registry

Job description

Empower Retirement is seeking a Lead Data Engineer to guide the design, development, and optimization of scalable data pipelines and real-time ingestion platforms across enterprise data ecosystems. You will mentor engineers, collaborate with architects, and drive automation, governance, and performance improvements using Kafka, Spark, Flink, and cloud technologies.

This role emphasizes resilience, security, and data quality, with a broad remit across streaming, lakehouse formats, and schema