Senior Data Engineer: Architect Lakehouse Pipelines

Dynata; United States; Remote; USD 150,000 - 190,000; Full time

Job summary

Dynata is hiring a Senior Data Engineer to design, build, and maintain data pipelines powering the enterprise lakehouse. You will deliver robust ELT/ETL solutions and mentor junior engineers.

You will design scalable batch and streaming pipelines using Spark, Kafka, and Flink; implement Bronze/Silver/Gold architecture with Delta Lake or Iceberg; and optimize PySpark transformations for large datasets.

Qualifications

• 6+ years of data engineering experience with a track record of enterprise-scale delivery.
• Expert proficiency in Python and SQL; PySpark experience required.
• Hands-on experience with Apache Spark, Delta Lake, or Apache Iceberg.
• Experience with orchestration tools: Apache Airflow, Prefect, or Dagster.
• Strong knowledge of cloud data services: AWS Glue, Azure Data Factory, GCP Dataflow.
• Proficiency with version control (Git), CI/CD pipelines, and containerization (Docker/Kubernetes).
• Experience with dbt (data build tool) for transformation layer management.
• Bachelor's degree in Computer Science, Engineering, or related technical field.

Responsibilities

• Design scalable batch and streaming data pipelines using Spark, Kafka, and Flink.
• Build and maintain Bronze/Silver/Gold medallion architecture within the lakehouse (Delta Lake / Iceberg).
• Develop and optimize complex SQL and PySpark transformations for large-scale datasets.
• Integrate structured, semi-structured, and unstructured data sources into the lakehouse.
• Collaborate with data architects to evolve data models.
• Implement data quality checks and monitoring with Great Expectations or dbt tests.
• Write Infrastructure-as-Code for pipelines (Terraform, Helm).
• Participate in code reviews and enforce engineering standards.
• Troubleshoot pipeline failures, performance bottlenecks, and data incidents.
• Mentor junior and mid-level data engineers and share knowledge.

Skills

Python

SQL

PySpark

Apache Spark

Delta Lake

Apache Iceberg

Airflow

Prefect

Dagster

AWS Glue

Azure Data Factory

GCP Dataflow

Git

CI/CD

Docker

Kubernetes

dbt

Databricks

Snowflake

Education

Bachelor's degree

Tools

Git

Docker

Kubernetes

Apache Spark

Delta Lake

Apache Iceberg

Airflow

Prefect

Dagster

dbt

Databricks

Snowflake

Job description

Dynata is hiring a Senior Data Engineer to design, build, and maintain data pipelines powering the enterprise lakehouse. You will deliver robust ELT/ETL solutions and mentor junior engineers.

You will design scalable batch and streaming pipelines using Spark, Kafka, and Flink; implement Bronze/Silver/Gold architecture with Delta Lake or Iceberg; and optimize PySpark transformations for large datasets.