Sr Data Engineer

Horizontal Talent; Brooklyn Park (MN); Presencial; USD 95.000 - 147.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Medical, dental, vision

Retirement plan

Descripción de la vacante

Horizontal Talent is seeking a Senior Data Engineer to build, modernize, and support large-scale data pipelines across Hadoop and Google Cloud Platform. This hands-on contract role focuses on reliable data delivery for analytics, fraud investigation, and ML use cases.

You will design and maintain scalable pipelines with Python, SQL, Spark, and Kafka, migrate workloads to GCP, and implement CI/CD for production data systems while collaborating with engineers and product partners.

Formación

• 5+ years of hands-on data engineering experience.
• Strong Python and SQL proficiency.
• Experience with Spark and distributed processing.
• Knowledge of Hadoop (HDFS, Hive).
• Experience with Kafka or similar.
• Experience with ETL/ELT pipelines.
• Experience with GCP, especially BigQuery.
• Experience with monitoring, logging, and incident troubleshooting.
• Familiarity with Git, CI/CD, automated testing, and Agile.

Responsabilidades

• Design, build, test, and maintain scalable batch and streaming data pipelines.
• Migrate and modernize data workloads from on-premises Hadoop to GCP.
• Develop and optimize ETL/ELT workflows for large, high-volume datasets.
• Create and support trusted datasets for analytics, reporting, fraud analysis, and ML.
• Investigate production data issues and perform root-cause analysis.
• Build monitoring, alerting, automation, and CI/CD for data systems.
• Collaborate with engineers, analysts, data scientists, and product partners.
• Contribute to code reviews, technical design discussions, documentation, and knowledge sharing.

Conocimientos

Data engineering

Python

SQL

Spark

Hadoop

Kafka

ETL/ELT pipelines

BigQuery

Monitoring/logging

CI/CD

Agile

Independent work

Herramientas

Apache Spark

Hadoop (HDFS/Hive)

Kafka

BigQuery

Airflow / Cloud Composer / Oozie

Git

CI/CD

Automated testing

Descripción del empleo

We are looking for a Senior Data Engineer to help build, modernize, and support large-scale data pipelines across Hadoop and Google Cloud Platform. This hands-on contract role is a great fit for someone who enjoys solving complex data challenges, improving platform reliability, and delivering trusted datasets for analytics, fraud investigation, and machine learning use cases.

Responsibilities

• Design, build, test, and maintain scalable batch and streaming data pipelines using Python, SQL, Spark, Kafka, and related tools.
• Support the migration and modernization of data workloads from on-premises Hadoop environments to GCP.
• Develop and optimize ETL/ELT workflows for large, high-volume datasets with attention to performance, reliability, and cost efficiency.
• Create and support trusted datasets that enable analytics, reporting, fraud analysis, and machine learning initiatives.
• Investigate production data issues, perform root-cause analysis, and implement durable fixes.
• Build monitoring, alerting, automation, and CI/CD practices that support stable production data systems.
• Collaborate with engineers, analysts, data scientists, and product partners to deliver timely data solutions.
• Contribute to code reviews, technical design discussions, documentation, and knowledge sharing.

Skills

• 5+ years of hands-on experience in data engineering or a related software engineering role building large-scale data systems.
• Strong proficiency in Python and SQL.
• Hands-on experience with Apache Spark and distributed data processing.
• Working knowledge of Hadoop technologies, including HDFS and Hive.
• Experience with Kafka or other event-streaming platforms.
• Experience building and supporting ETL/ELT and data integration pipelines.
• Experience with GCP data services, especially BigQuery.
• Experience supporting production data systems, including monitoring, logging, and incident troubleshooting.
• Familiarity with Git, CI/CD, automated testing, and Agile ways of working.
• Strong problem-solving skills and the ability to work independently within an established architecture.

Preferred Skills

• Experience migrating Hadoop workloads or enterprise data platforms to GCP or another public cloud.
• Experience with Dataproc, Dataflow, Pub/Sub, or Cloud Storage.
• Experience with Spark Streaming, Flink, or similar stream-processing frameworks.
• Exposure to data governance, lineage, privacy, security, and data quality frameworks.
• Experience with orchestration tools such as Airflow, Cloud Composer, or Oozie.
• Experience supporting ML data pipelines or Vertex AI / MLOps workflows.
• Background working with high-volume enterprise data in fraud, risk, security, retail, or e-commerce environments.

Horizontal Talent is committed to diversity, equity, and inclusion, and we welcome candidates from all backgrounds and experiences. We believe inclusive teams create stronger solutions, and we are dedicated to fostering a respectful and supportive hiring experience for everyone.

Applications will be accepted for 4 weeks.

For those that join the team, we offer competitive compensation and benefits including medical, dental, vision, and retirement.

• medical, dental, vision, and retirement.

The pay range for this role is $69 - $107 per hour based on qualifications and experience.