Lead Data Platform Engineer

Elasticsearch B.V.; Northern (KY); Remote; USD 113,000 - 179,000; Full time

Benefits offered by this job

Health coverage

Flexible locations & schedules

Generous vacation days

Donation matching

Volunteer time

Job summary

Elastic is seeking a Lead Data Engineer to join the Data Engineering & Architecture team to grow a data platform that enables teams across the company to work with data confidently and effectively. You will design, optimize, and expand pipelines powering ingestion, streaming, transformation, and delivery, using Airflow, Spark, Iceberg, Kafka, Flink, and dbt Labs.

You will engage with stakeholders to define data flows, co-create data contracts, author documentation, and mentor teammates to

Qualifications

• 8+ years of experience in data engineering or related roles.
• Experience with Apache Airflow for orchestrating data workflows.
• Experience with Spark, Flink, Kafka, and dbt Labs for data processing and transformation.
• Understanding of Apache Iceberg for data table formats and data warehouse/lakehouse strategies.
• Skilled in using Terraform for infrastructure-as-code deployments.
• Proficient in Bash, Python, and SQL.
• Knowledge of OLAP, OLTP, Document, and Vector databases.
• Experience with Google Cloud Platform (GCP) for cloud resource management.
• Bonus: Experience with data security and access controls for SOX compliance.
• Bonus: Experience with Elastic stack.

Responsibilities

• Design, maintain, and improve pipelines and infrastructure powering our data platform.
• Work across the full data lifecycle from ingestion to delivery using Airflow, Spark, Iceberg, Kafka, Flink, and dbt Labs.
• Collaborate with stakeholders, co-create data contracts, document, and guide teammates on platform usage.
• Monitor and refine pipeline performance and ensure robust processes.
• Use Terraform to provision and manage resources on GCP.
• Review code and promote best practices and standard processes.
• Handle data access requests and ensure compliance with platform policies.

Skills

Data engineering

Apache Airflow

Spark

Flink

Kafka

dbt Labs

Terraform

Bash

Python

SQL

GCP

OLAP/OLTP

Elastic stack

Data security

Tools

Google Cloud Platform (GCP)

Apache Iceberg

Job description

Elastic is seeking a Lead Data Engineer to join the Data Engineering & Architecture team to grow a data platform that enables teams across the company to work with data confidently and effectively. You will design, optimize, and expand pipelines powering ingestion, streaming, transformation, and delivery, using Airflow, Spark, Iceberg, Kafka, Flink, and dbt Labs.

You will engage with stakeholders to define data flows, co-create data contracts, author documentation, and mentor teammates to