Lead Data Engineer: Build Scalable Data Platform

Carbon Data Solutions; United States; On-site; USD 113,000 - 179,000; Full time

Benefits offered by this job

Stock program

401k matching up to 6%

Health coverage

Job summary

Elastic is seeking a Lead Data Engineer to join the Data Engineering & Architecture team. You will design, maintain, and improve the pipelines and infrastructure powering the data platform from ingestion to delivery, using Airflow, Spark, Iceberg, Kafka, Flink, and dbt Labs.

You’ll collaborate with partners, define data contracts, and mentor teammates. Awarded stock programs and a strong total rewards package are offered, including a 401(k) with 6% match and comprehensive health coverage.

Qualifications

• 8+ years of experience in data engineering or related roles.
• Experience with Apache Airflow for orchestrating data workflows.
• Experience with Spark, Flink, Kafka, and dbt Labs for data processing and transformation.
• Understanding of Apache Iceberg for data table formats and data warehouse/lakehouse strategies.
• Skilled in using Terraform for infrastructure-as-code deployments.
• Proficient in Bash, Python, and SQL.
• Knowledge of database systems including OLAP/OLTP/Document/Vector databases.
• Experience with Google Cloud Platform (GCP) for cloud resource management.
• Bonus: Data security and access controls for SOX compliance.
• Bonus: Experience/knowledge of the Elastic stack.

Responsibilities

• Design, maintain, and improve pipelines and infrastructure powering the data platform.
• Handle full data lifecycle from ingestion to delivery using modern tools.
• Collaborate with stakeholders to define data flows and data contracts; write documentation.
• Review code and promote good practices; support platform governance.
• Manage access requests and ensure compliance with platform policies.

Skills

Apache Airflow

Spark

Flink

Kafka

dbt Labs

Iceberg

Terraform

Bash

Python

SQL

GCP

SOX Compliance

Elastic stack

Tools

Airflow

Spark

Iceberg

Kafka

Flink

dbt Labs

Job description

Elastic is seeking a Lead Data Engineer to join the Data Engineering & Architecture team. You will design, maintain, and improve the pipelines and infrastructure powering the data platform from ingestion to delivery, using Airflow, Spark, Iceberg, Kafka, Flink, and dbt Labs.

You’ll collaborate with partners, define data contracts, and mentor teammates. Awarded stock programs and a strong total rewards package are offered, including a 401(k) with 6% match and comprehensive health coverage.