Senior Data Engineer: Scale Data Pipelines on GCP

Horizontal Talent; Brooklyn Park (MN); On-site; USD 95,000 - 147,000; Full time

Benefits offered by this job

Medical, dental, vision

Retirement plan

Job summary

Horizontal Talent is seeking a Senior Data Engineer to build, modernize, and support large-scale data pipelines across Hadoop and Google Cloud Platform. This hands-on contract role focuses on reliable data delivery for analytics, fraud investigation, and ML use cases.

You will design and maintain scalable pipelines with Python, SQL, Spark, and Kafka, migrate workloads to GCP, and implement CI/CD for production data systems while collaborating with engineers and product partners.

Qualifications

• 5+ years of hands-on data engineering experience.
• Strong Python and SQL proficiency.
• Experience with Spark and distributed processing.
• Knowledge of Hadoop (HDFS, Hive).
• Experience with Kafka or similar.
• Experience with ETL/ELT pipelines.
• Experience with GCP, especially BigQuery.
• Experience with monitoring, logging, and incident troubleshooting.
• Familiarity with Git, CI/CD, automated testing, and Agile.

Responsibilities

• Design, build, test, and maintain scalable batch and streaming data pipelines.
• Migrate and modernize data workloads from on-premises Hadoop to GCP.
• Develop and optimize ETL/ELT workflows for large, high-volume datasets.
• Create and support trusted datasets for analytics, reporting, fraud analysis, and ML.
• Investigate production data issues and perform root-cause analysis.
• Build monitoring, alerting, automation, and CI/CD for data systems.
• Collaborate with engineers, analysts, data scientists, and product partners.
• Contribute to code reviews, technical design discussions, documentation, and knowledge sharing.

Skills

Data engineering

Python

SQL

Spark

Hadoop

Kafka

ETL/ELT pipelines

BigQuery

Monitoring/logging

CI/CD

Agile

Independent work

Tools

Apache Spark

Hadoop (HDFS/Hive)

Kafka

BigQuery

Airflow / Cloud Composer / Oozie

Git

CI/CD

Automated testing

Job description

Horizontal Talent is seeking a Senior Data Engineer to build, modernize, and support large-scale data pipelines across Hadoop and Google Cloud Platform. This hands-on contract role focuses on reliable data delivery for analytics, fraud investigation, and ML use cases.

You will design and maintain scalable pipelines with Python, SQL, Spark, and Kafka, migrate workloads to GCP, and implement CI/CD for production data systems while collaborating with engineers and product partners.