Lead Data Engineer — Build Scalable Data Platform

Elastic; United States; Hybrid; USD 113,000 - 179,000; Full time

Benefits offered by this job

Stock program

401k with match

Health coverage

Flexible location

Generous vacation

Volunteer hours

Parental leave

Job summary

Elastic, the Search AI Company, is seeking a Lead Data Engineer to grow our Data Platform. You’ll design and optimize pipelines across ingestion, streaming, transformation, and delivery, using Airflow, Spark, Iceberg, Kafka, Flink, and dbt.

Collaborate with partners, share knowledge, and empower teams to trust data. You’ll work with stakeholders to shape data contracts, write documentation, and mentor teammates.

Qualifications

• 8+ years of experience in data engineering or related roles.
• Experience with Apache Airflow for orchestrating data workflows.
• Experience with Spark, Flink, Kafka, and dbt Labs for data processing and transformation.
• Understanding of Apache Iceberg for data table formats and data warehouse/lakehouse strategies.
• Skilled in using Terraform for infrastructure-as-code deployments.
• Proficient in Bash, Python, and SQL.
• Knowledge of OLAP, OLTP, Document, and Vector databases.
• Experience with Google Cloud Platform (GCP).
• Bonus: Data security and access controls for SOX compliance.
• Bonus: Experience with Elastic stack.

Responsibilities

• Design, maintain, and improve pipelines and infrastructure powering the data platform.
• Work across the full data lifecycle from ingestion to delivery using Airflow, Spark, Iceberg, Kafka, Flink, and dbt.
• Collaborate with stakeholders to define data flows and write clear documentation.
• Review code and promote best practices; manage access requests and compliance.

Skills

Apache Airflow

Spark

Flink

Kafka

dbt Labs

Terraform

Python

SQL

GCP

Elastic Stack

Tools

GCP

Elastic Stack

Iceberg

Job description

Elastic, the Search AI Company, is seeking a Lead Data Engineer to grow our Data Platform. You’ll design and optimize pipelines across ingestion, streaming, transformation, and delivery, using Airflow, Spark, Iceberg, Kafka, Flink, and dbt.

Collaborate with partners, share knowledge, and empower teams to trust data. You’ll work with stakeholders to shape data contracts, write documentation, and mentor teammates.