Senior Data Engineer (Hybrid) - GCP Spark Pipelines

Dahl Consulting; Brooklyn Park (MN); Hybride; USD 92 000 - 156 000; Temps partiel

Résumé du poste

Dahl Consulting is seeking a Senior Data Engineer in Brooklyn Park, MN on a hybrid contract (6-12 months) to build and modernize large-scale data pipelines using Python, SQL, Spark, and Kafka. This role supports a fraud and abuse tech environment within a national retailer's data services engineering team.

Responsibilities include migrating pipelines to Google Cloud Platform, ensuring data quality, and delivering scalable data solutions for high-volume batch and streaming workloads.

Qualifications

• 5+ years of hands-on data engineering or related software engineering experience.
• Strong proficiency in Python and SQL.
• Hands-on experience with Apache Spark and distributed data processing.
• Experience with the Hadoop ecosystem, including HDFS and Hive.
• Hands-on experience with Kafka or another event-streaming technology.
• Experience building and supporting ETL/ELT and data integration pipelines.

Responsabilités

• Design, build, test, deploy, and support scalable batch and streaming data pipelines using Python, SQL, Apache Spark, Kafka, and related technologies.
• Support the modernization and migration of data pipelines and datasets from on-premises Hadoop environments to Google Cloud Platform (GCP), including BigQuery and other cloud-native services.
• Build and optimize ETL/ELT workflows for large-scale, high-volume datasets with an emphasis on performance, scalability, reliability, and cost efficiency.
• Develop trusted datasets supporting analytics, fraud investigation, reporting, and machine learning use cases.
• Implement appropriate data quality checks, testing, and validation processes.
• Troubleshoot production data issues, conduct root-cause analysis, and implement durable solutions to prevent recurring failures.
• Implement and maintain monitoring, alerting, automation, and CI/CD practices for production data pipelines.
• Work across both legacy Hadoop and modern GCP environments during ongoing platform modernization.
• Partner with engineers, analysts, data scientists, product teams, and other stakeholders to understand data requirements and deliver effective solutions.
• Identify technical risks, data quality concerns, performance issues, and opportunities to improve platform reliability.
• Contribute to technical design, code reviews, engineering standards, documentation, and knowledge transfer.

Connaissances

Python

SQL

Apache Spark

Kafka

Hadoop

BigQuery

Data pipelines

CI/CD

Git

Agile

Outils

HDFS

Hive

Airflow

Dataproc

Dataflow

Cloud Composer

Description du poste

Dahl Consulting is seeking a Senior Data Engineer in Brooklyn Park, MN on a hybrid contract (6-12 months) to build and modernize large-scale data pipelines using Python, SQL, Spark, and Kafka. This role supports a fraud and abuse tech environment within a national retailer's data services engineering team.

Responsibilities include migrating pipelines to Google Cloud Platform, ensuring data quality, and delivering scalable data solutions for high-volume batch and streaming workloads.