Senior Data Engineer

Dynata; United States; Remote; USD 150,000 - 190,000; Full time

Job summary

Dynata is hiring a Senior Data Engineer to design, build, and maintain data pipelines powering the enterprise lakehouse. You will deliver robust ELT/ETL solutions and mentor junior engineers.

You will design scalable batch and streaming pipelines using Spark, Kafka, and Flink; implement Bronze/Silver/Gold architecture with Delta Lake or Iceberg; and optimize PySpark transformations for large datasets.

Qualifications

• 6+ years of data engineering experience with a track record of enterprise-scale delivery.
• Expert proficiency in Python and SQL; PySpark experience required.
• Hands-on experience with Apache Spark, Delta Lake, or Apache Iceberg.
• Experience with orchestration tools: Apache Airflow, Prefect, or Dagster.
• Strong knowledge of cloud data services: AWS Glue, Azure Data Factory, GCP Dataflow.
• Proficiency with version control (Git), CI/CD pipelines, and containerization (Docker/Kubernetes).
• Experience with dbt (data build tool) for transformation layer management.
• Bachelor's degree in Computer Science, Engineering, or related technical field.

Responsibilities

• Design scalable batch and streaming data pipelines using Spark, Kafka, and Flink.
• Build and maintain Bronze/Silver/Gold medallion architecture within the lakehouse (Delta Lake / Iceberg).
• Develop and optimize complex SQL and PySpark transformations for large-scale datasets.
• Integrate structured, semi-structured, and unstructured data sources into the lakehouse.
• Collaborate with data architects to evolve data models.
• Implement data quality checks and monitoring with Great Expectations or dbt tests.
• Write Infrastructure-as-Code for pipelines (Terraform, Helm).
• Participate in code reviews and enforce engineering standards.
• Troubleshoot pipeline failures, performance bottlenecks, and data incidents.
• Mentor junior and mid-level data engineers and share knowledge.

Skills

Python

SQL

PySpark

Apache Spark

Delta Lake

Apache Iceberg

Airflow

Prefect

Dagster

AWS Glue

Azure Data Factory

GCP Dataflow

Git

CI/CD

Docker

Kubernetes

dbt

Databricks

Snowflake

Education

Bachelor's degree

Tools

Git

Docker

Kubernetes

Apache Spark

Delta Lake

Apache Iceberg

Airflow

Prefect

Dagster

dbt

Databricks

Snowflake

Job description

The Senior Data Engineer is responsible for designing, building, and maintaining the data pipelines, transformation layers, and data models that power the enterprise lakehouse. This role is a technical anchor on the data engineering team, delivering robust ELT/ETL solutions and serving as a mentor to junior engineers.

KEY RESPONSIBILITIES

• Design and implement scalable batch and streaming data pipelines using Apache Spark, Kafka, and Flink
• Build and maintain the Bronze/Silver/Gold medallion architecture within the lakehouse (Delta Lake / Iceberg)
• Develop and optimize complex SQL and PySpark transformations for large-scale datasets
• Integrate structured, semi-structured, and unstructured data sources into the lakehouse
• Collaborate with data architects to evolve the physical and logical data models
• Implement data quality checks and monitoring using Great Expectations or dbt tests
• Write Infrastructure-as-Code for pipeline environments (Terraform, Helm)
• Participate in code reviews and enforce engineering standards and best practices
• Troubleshoot pipeline failures, performance bottlenecks, and data incidents
• Mentor junior and mid-level data engineers and contribute to internal knowledge sharing

REQUIRED QUALIFICATIONS

• 6+ years of data engineering experience with a track record of enterprise-scale delivery
• Expert proficiency in Python and SQL; PySpark experience required
• Hands-on experience with Apache Spark, Delta Lake, or Apache Iceberg
• Experience with orchestration tools: Apache Airflow, Prefect, or Dagster
• Strong knowledge of cloud data services: AWS Glue, Azure Data Factory, GCP Dataflow
• Proficiency with version control (Git), CI/CD pipelines, and containerization (Docker/Kubernetes)
• Experience with dbt (data build tool) for transformation layer management
• Bachelor's degree in Computer Science, Engineering, or related technical field

PREFERRED QUALIFICATIONS

• Experience with Databricks, Snowflake, or Apache Hudi
• Knowledge of streaming architectures and Apache Kafka
• Certifications: Databricks Certified Data Engineer, AWS Data Analytics Specialty

At Dynata, we are committed to fostering an inclusive, accessible environment, where all employees and customers feel valued, respected and supported. We are dedicated to building a workforce that reflects the diversity of our customers and communities in which we live and serve. Dynata welcomes and encourages applications from people with disabilities. We are committed to an inclusive work culture for all our employees. Accommodations by request can be made for all aspects of the selection process.