Lead Data Platform Engineer - Streaming & Lakehouse

Accion Labs; United States; On-site; USD 180,000 - 260,000; Full time

Job summary

Accion Labs is seeking a hands-on Principal/Lead Data Engineer to design and evolve the data platform across ingestion, stream processing, lakehouse storage, and serving layers.

You will architect event-driven pipelines using Kafka and Flink, implement CDC with Debezium and Qlik Replicate, and develop production pipelines in Databricks with Python and Spark. You will mentor engineers and set reusable patterns across teams.

Qualifications

• Minimum 8+ years of software or data-engineering experience.
• Strong Apache Kafka expertise, including topic design and troubleshooting.
• Hands-on Apache Flink experience with stateful processing and recovery.
• Strong CDC experience with database change capture and schema evolution.
• Hands-on Apache Iceberg/Delta table design and maintenance.
• Strong Databricks and Apache Spark production pipeline experience.
• Proficient in Python and SQL for maintainable data workflows.
• Experience building data platforms on Azure, including secure connectivity.

Responsibilities

• Build streaming data pipelines with Kafka and Flink, including stateful processing and enrichments.
• Own CDC integration using Qlik Replicate and Debezium; manage snapshots and updates.
• Develop the lakehouse with Iceberg/Delta storage and schema evolution.
• Engineer production pipelines on Databricks using Python, SQL, and Spark.
• Ensure data quality, lineage, and access controls across pipelines.
• Automate delivery with CI/CD, IaC, and repeatable deployments.
• Operate production systems: monitor freshness, lag, throughput, and failures.
• Lead design reviews, mentor engineers, and set reusable implementation patterns.

Skills

Data engineering leadership

Apache Kafka

Apache Flink

CDC tooling

Apache Iceberg

Delta Lake

Databricks

Python

SQL

Spark

Azure cloud

Kubernetes

Tools

Qlik Replicate

Debezium

Kubernetes

Job description

Accion Labs is seeking a hands-on Principal/Lead Data Engineer to design and evolve the data platform across ingestion, stream processing, lakehouse storage, and serving layers.

You will architect event-driven pipelines using Kafka and Flink, implement CDC with Debezium and Qlik Replicate, and develop production pipelines in Databricks with Python and Spark. You will mentor engineers and set reusable patterns across teams.