Cloud Data Engineer: Spark, ETL & BigQuery (Hybrid)

Horizontal Talent; Brooklyn Park (MN); Hybrid; USD 51,000 - 98,000; Full time

Job summary

Horizontal is seeking a Data Engineer to design and deliver scalable data solutions powering analytics and decision-making. This hybrid role offers collaboration on cloud data platforms, large-scale processing, and reliable pipelines.

You will build ETL/ELT pipelines with Spark and GCP services, optimize BigQuery, and partner with architects and stakeholders to productionize data products with DevOps practices.

Qualifications

• Relevant experience building production data pipelines in a data engineering environment.
• Hands-on ETL/ELT development experience.
• Practical experience with Google Cloud Platform and BigQuery.
• Strong SQL skills and the ability to work with large datasets.
• Experience with Apache Spark and distributed data processing.
• Exposure to Hadoop and Kafka is a plus.
• Proficiency in Python and/or Java/Scala.
• Understanding of data modeling, partitioning, and common data formats.
• Experience supporting production systems and implementing DevOps practices (Git, CI/CD).

Responsibilities

• Design, build, test, and maintain scalable ETL/ELT pipelines for large datasets.
• Develop distributed data processing solutions using Spark and Google Cloud services.
• Create and support analytical data assets in BigQuery with performance and cost focus.
• Build batch and streaming data workflows using Spark, Hadoop, and Kafka.
• Develop ingestion and transformation processes across source systems and data lakes.
• Troubleshoot production data issues and implement durable fixes.
• Implement data quality, monitoring, logging, and alerting for stable operations.
• Collaborate with architects, engineers, and stakeholders to productionize data products.
• Contribute to code reviews, reusable frameworks, and engineering best practices.
• Apply DevOps practices including version control, automated testing, CI/CD, and deployment automation.

Skills

ETL/ELT pipelines

Python

SQL

Google Cloud Platform

BigQuery

Apache Spark

Kafka

Java/Scala

Looker/LookML

Terraform

Tools

GCP Dataproc

Job description

Horizontal is seeking a Data Engineer to design and deliver scalable data solutions powering analytics and decision-making. This hybrid role offers collaboration on cloud data platforms, large-scale processing, and reliable pipelines.

You will build ETL/ELT pipelines with Spark and GCP services, optimize BigQuery, and partner with architects and stakeholders to productionize data products with DevOps practices.