Senior Data Engineer - Remote (GCP, Spark/Kafka)

Dahl Consulting; Brooklyn Park (MN); Hybrid; USD 92,000 - 156,000; Full time

Job summary

Dahl Consulting is seeking a Senior Data Engineer for a national retailer with a sophisticated tech organization. This remote contract role focuses on building and operating large-scale data pipelines across Hadoop and GCP, delivering reliable batch and streaming workloads.

The ideal candidate will own data engineering work from design through production, collaborate with engineers, analysts, and data scientists, and optimize pipelines for performance, cost, and reliability in a fast-paced

Qualifications

• 5+ years of hands-on data engineering or related software engineering experience building large-scale data systems.
• Strong proficiency in Python and SQL.
• Hands-on experience with Apache Spark and distributed data processing.
• Experience with the Hadoop ecosystem, including HDFS and Hive.
• Hands-on experience with Kafka or another event-streaming technology.
• Experience building and supporting ETL/ELT and data integration pipelines.
• Experience with Google Cloud Platform data services, particularly BigQuery.
• Experience optimizing data pipelines for performance, scalability, reliability, and cost.
• Experience supporting production data systems, including monitoring, logging, incident troubleshooting, and operational support.
• Working knowledge of Git, CI/CD, automated testing, and Agile engineering practices.
• Ability to independently own data engineering work from design through production deployment and ongoing support.
• Strong communication and collaboration skills with the ability to work effectively across technical and business teams.

Responsibilities

• Design, build, test, deploy, and support scalable batch and streaming data pipelines using Python, SQL, Apache Spark, Kafka, and related technologies.
• Support the modernization and migration of data pipelines and datasets from on-premises Hadoop environments to Google Cloud Platform (GCP), including BigQuery and other cloud-native services.
• Build and optimize ETL/ELT workflows for large-scale, high-volume datasets with an emphasis on performance, scalability, reliability, and cost efficiency.
• Develop trusted datasets supporting analytics, fraud investigation, reporting, and machine learning use cases.
• Implement appropriate data quality checks, testing, and validation processes.
• Troubleshoot production data issues, conduct root-cause analysis, and implement durable solutions to prevent recurring failures.
• Implement and maintain monitoring, alerting, automation, and CI/CD practices for production data pipelines.
• Work across both legacy Hadoop and modern GCP environments during ongoing platform modernization.
• Partner with engineers, analysts, data scientists, product teams, and other stakeholders to understand data requirements and deliver effective solutions.
• Identify technical risks, data quality concerns, performance issues, and opportunities to improve platform reliability.
• Contribute to technical design, code reviews, engineering standards, documentation, and knowledge transfer.

Skills

Python

SQL

Apache Spark

Kafka

Hadoop

Hive

BigQuery

Dataproc

CI/CD

Git

Airflow

Data pipelines

Tools

Hadoop ecosystem

Hive

BigQuery

Dataproc

Dataflow

Pub/Sub

Cloud Storage

Airflow

Cloud Composer

Oozie

Spark Streaming

Flink

Job description

Dahl Consulting is seeking a Senior Data Engineer for a national retailer with a sophisticated tech organization. This remote contract role focuses on building and operating large-scale data pipelines across Hadoop and GCP, delivering reliable batch and streaming workloads.

The ideal candidate will own data engineering work from design through production, collaborate with engineers, analysts, and data scientists, and optimize pipelines for performance, cost, and reliability in a fast-paced