Principal/Lead Data Engineer

Accion Labs; United States; On-site; USD 180,000 - 260,000; Full time

Job summary

Accion Labs is seeking a hands-on Principal/Lead Data Engineer to design and evolve the data platform across ingestion, stream processing, lakehouse storage, and serving layers.

You will architect event-driven pipelines using Kafka and Flink, implement CDC with Debezium and Qlik Replicate, and develop production pipelines in Databricks with Python and Spark. You will mentor engineers and set reusable patterns across teams.

Qualifications

• Minimum 8+ years of software or data-engineering experience.
• Strong Apache Kafka expertise, including topic design and troubleshooting.
• Hands-on Apache Flink experience with stateful processing and recovery.
• Strong CDC experience with database change capture and schema evolution.
• Hands-on Apache Iceberg/Delta table design and maintenance.
• Strong Databricks and Apache Spark production pipeline experience.
• Proficient in Python and SQL for maintainable data workflows.
• Experience building data platforms on Azure, including secure connectivity.

Responsibilities

• Build streaming data pipelines with Kafka and Flink, including stateful processing and enrichments.
• Own CDC integration using Qlik Replicate and Debezium; manage snapshots and updates.
• Develop the lakehouse with Iceberg/Delta storage and schema evolution.
• Engineer production pipelines on Databricks using Python, SQL, and Spark.
• Ensure data quality, lineage, and access controls across pipelines.
• Automate delivery with CI/CD, IaC, and repeatable deployments.
• Operate production systems: monitor freshness, lag, throughput, and failures.
• Lead design reviews, mentor engineers, and set reusable implementation patterns.

Skills

Data engineering leadership

Apache Kafka

Apache Flink

CDC tooling

Apache Iceberg

Delta Lake

Databricks

Python

SQL

Spark

Azure cloud

Kubernetes

Tools

Qlik Replicate

Debezium

Kubernetes

Job description

We are looking for a hands‑on Principal / Lead Data Engineer to build and evolve the data platform. You will design and deliver batch and streaming pipelines using Kafka, Flink, CDC, Apache Iceberg, and Databricks, making trusted data available for applications, analytics, and AI.

This role combines strong software engineering with deep data‑engineering expertise. You will own capabilities across ingestion, stream processing, lakehouse storage, transformation, and serving—including deployment, monitoring, troubleshooting, and recovery. You will guide architectural decisions, mentor engineers, and establish reusable patterns across teams.

What You Will Do

• Build streaming data pipelines: Design and implement event‑driven pipelines using Apache Kafka and Apache Flink, including stateful processing, enrichment, joins, and aggregations.
• Own CDC integration: Capture changes from operational databases using tools such as Qlik Replicate and Debezium. Handle initial snapshots, incremental updates, deletes, schema changes, and recovery.
• Develop the lakehouse: Build and maintain Apache Iceberg and Delta Lake datasets, defining table structure, partitioning, schema evolution, maintenance, and retention strategies.
• Engineer on Databricks: Develop production pipelines using Python, SQL, Spark, and Databricks for ingestion, transformation, reconciliation, and curated data delivery.
• Ensure data correctness: Design for duplicate and late‑arriving events, event ordering, idempotency, replay, and backfills. Validate processing guarantees across the complete pipeline. Design reusable data models: Translate source‑system structures and business requirements into models supporting historical analysis, reliable joins, and application consumption.
• Deliver governed data: Establish data quality, metadata, lineage, ownership, and access controls, including tenant isolation and protection of sensitive data.
• Automate delivery: Build automated tests, CI/CD pipelines, infrastructure as code, and repeatable deployments across environments.
• Operate production systems: Monitor freshness, completeness, lag, throughput, and failures. Troubleshoot issues across connectors, brokers, processing jobs, and storage.
• Optimize performance and cost: Tune streaming and batch workloads, resource allocation, file sizes, partitioning, and query performance.
• Provide technical leadership: Lead design and code reviews, mentor engineers, and establish practical standards and reusable implementation patterns.

What You Bring

• 8+ years of relevant software or data‑engineering experience, including technical leadership for complex production data systems.
• Strong Apache Kafka expertise, including topic and partition design, consumer groups, offsets, message schemas, delivery semantics, and operational troubleshooting.
• Hands‑on Apache Flink experience, including stateful stream processing, event time, watermarks, checkpoints, savepoints, recovery, and performance tuning.
• Strong CDC experience, including database change capture, snapshot‑to‑stream transitions, updates and deletes, schema evolution, reconciliation, and replay.
• Hands‑on Apache Iceberg/Delta Table experience, including table design, schema and partition evolution, snapshots, compaction, and integration with processing engines and catalogs.
• Strong Databricks and Apache Spark experience, including production pipeline development, orchestration, distributed processing, troubleshooting, and optimization.
• Strong Python and SQL skills, with experience building maintainable, tested code beyond exploratory notebooks.
• Experience building data platforms on Azure, including storage, identity, networking, and secure connectivity.
• Java experience for stream‑processing applications and connector extensions.
• Experience implementing automated data‑quality checks, source‑to‑target reconciliation, lineage, and access controls.
• Practical experience with CI/CD, automated testing, and infrastructure as code for data workloads.
• Working experience deploying and troubleshooting containerized data services on Kubernetes.
• Ability to explain technical trade‑offs, collaborate with domain experts, and guide engineers through implementation and production support.

Nice to Have

• Experience with Confluent Cloud, Schema Registry, Kafka Connect, and Tableflow.
• Experience with Qlik Replicate, Debezium, or custom connector development.
• Experience with Databricks Lakeflow, Unity Catalog, Delta Lake, and Delta Sharing.
• Familiarity with DataHub or similar metadata and lineage platforms.
• Experience serving curated data through PostgreSQL, Databricks Lakebase, or APIs.
• Experience preparing governed data for AI applications, semantic models, or entity resolution.