Senior Data Engineer: Real-Time Pipelines Lead (Remote)

Empower; United States; Remote; USD 132,000 - 191,000; Full time

Benefits offered by this job

Medical, dental, vision

401(k) with company match

Tuition reimbursement

Paid time off

Volunteer time

BRGs

Casual dress

Job summary

Empower is seeking a Lead Data Engineer to architect and optimize scalable data pipelines, real-time ingestion, and enterprise data platforms. You will lead complex data initiatives, set standards for performance, security, and governance, and mentor engineers in a highly independent role.

You will design end-to-end streaming solutions, manage CDC and schema evolution, and collaborate with stakeholders to drive architectural decisions aligned with enterprise direction.

Qualifications

• Bachelor’s degree in computer science or related field or equivalent experience.
• 6-8+ years data engineering on enterprise-scale platforms.
• 6-8+ years hands-on with Apache Kafka, Confluent Cloud, AWS Kinesis, or Apache Pulsar.
• Advanced proficiency in Python, PySpark, and SQL.
• Proficiency in Python, Java, Scala, or Go with Flink/Kafka Streams/Spark.
• Strong AWS and modern data platforms experience.
• Deep expertise in CDC tools like Debezium and Kafka Connect.
• Experience with Iceberg/Delta Lake and Snowflake/BigQuery.
• Docker, Kubernetes, Terraform, and CI/CD for event-driven microservices.
• Data contracts and schema management with Schema Registry.
• Data modeling, ETL/ELT, and data warehousing knowledge.
• Led technical initiatives and mentored engineers.
• Strong problem-solving and communication; Agile experience.

Responsibilities

• Lead design and implementation of data pipelines, ingestion frameworks, and real-time streaming solutions.
• Build and maintain Kafka producers/consumers for sub-second latency across distributed services.
• Configure CDC solutions across relational/NoSQL systems into Kafka topics.
• Develop stream-processing with Flink, Spark Streaming, Kafka Streams, or dbt-mesh.
• Implement schema evolution with Confluent Schema Registry and ensure idempotency.
• Integrate automated generation, validation, and orchestration for schema drift and routing.
• Implement end-to-end pipeline monitoring with Prometheus, Grafana, or Datadog.
• Mentor engineers; lead code reviews and architectural decisions.
• Collaborate with architects to align with enterprise direction.
• Drive automation and platform capability improvements.
• Complete other duties as assigned.

Skills

Python

SQL

Java

Go

Scala

Apache Kafka

Distributed systems

AWS

Data pipelines

Streaming

Education

Bachelor's degree in computer science or related field

Tools

Apache Kafka

Confluent Cloud

AWS Kinesis

Apache Pulsar

Debezium

Kafka Connect

LangGraph

AutoGen

CrewAI

Schema Registry

Iceberg

Delta Lake

Snowflake

BigQuery

Docker

Kubernetes

Terraform

Job description

Empower is seeking a Lead Data Engineer to architect and optimize scalable data pipelines, real-time ingestion, and enterprise data platforms. You will lead complex data initiatives, set standards for performance, security, and governance, and mentor engineers in a highly independent role.

You will design end-to-end streaming solutions, manage CDC and schema evolution, and collaborate with stakeholders to drive architectural decisions aligned with enterprise direction.