Data Engineer

Horizontal Talent; Brooklyn Park (MN); Hybrid; USD 51,000 - 98,000; Full time

Job summary

Horizontal is seeking a Data Engineer to design and deliver scalable data solutions powering analytics and decision-making. This hybrid role offers collaboration on cloud data platforms, large-scale processing, and reliable pipelines.

You will build ETL/ELT pipelines with Spark and GCP services, optimize BigQuery, and partner with architects and stakeholders to productionize data products with DevOps practices.

Qualifications

• Relevant experience building production data pipelines in a data engineering environment.
• Hands-on ETL/ELT development experience.
• Practical experience with Google Cloud Platform and BigQuery.
• Strong SQL skills and the ability to work with large datasets.
• Experience with Apache Spark and distributed data processing.
• Exposure to Hadoop and Kafka is a plus.
• Proficiency in Python and/or Java/Scala.
• Understanding of data modeling, partitioning, and common data formats.
• Experience supporting production systems and implementing DevOps practices (Git, CI/CD).

Responsibilities

• Design, build, test, and maintain scalable ETL/ELT pipelines for large datasets.
• Develop distributed data processing solutions using Spark and Google Cloud services.
• Create and support analytical data assets in BigQuery with performance and cost focus.
• Build batch and streaming data workflows using Spark, Hadoop, and Kafka.
• Develop ingestion and transformation processes across source systems and data lakes.
• Troubleshoot production data issues and implement durable fixes.
• Implement data quality, monitoring, logging, and alerting for stable operations.
• Collaborate with architects, engineers, and stakeholders to productionize data products.
• Contribute to code reviews, reusable frameworks, and engineering best practices.
• Apply DevOps practices including version control, automated testing, CI/CD, and deployment automation.

Skills

ETL/ELT pipelines

Python

SQL

Google Cloud Platform

BigQuery

Apache Spark

Kafka

Java/Scala

Looker/LookML

Terraform

Tools

GCP Dataproc

Job description

We’re looking for a skilled Data Engineer to help design and deliver scalable data solutions that power analytics and business decision-making. This hybrid opportunity offers the chance to work with modern cloud data platforms, large-scale processing tools, and a collaborative team focused on building reliable, high-performing pipelines.

Responsibilities

• Design, build, test, and maintain scalable ETL and ELT pipelines for large and complex datasets.
• Develop distributed data processing solutions using Apache Spark and Google Cloud Platform services.
• Create and support analytical data assets in BigQuery, with a focus on performance, cost efficiency, and reliability.
• Build batch and streaming data workflows using technologies such as Spark, Hadoop, and Kafka.
• Develop ingestion and transformation processes across source systems, data lakes, and analytics environments.
• Troubleshoot production data issues, identify root causes, and implement durable fixes.
• Put data quality, monitoring, logging, and alerting controls in place to support stable operations.
• Partner with architects, engineers, and business stakeholders to turn requirements into production-ready solutions.
• Contribute to code reviews, reusable frameworks, and engineering best practices.
• Apply software engineering and DevOps practices, including version control, automated testing, CI/CD, and deployment automation.

Skills

• Strong experience building production data pipelines in a data engineering environment.
• Hands-on ETL/ELT development experience.
• Practical experience with Google Cloud Platform.
• Solid BigQuery skills, including SQL development and query optimization.
• Experience working with Apache Spark and distributed data processing.
• Experience with GCP Dataproc.
• Exposure to Hadoop and/or Kafka.
• Strong SQL skills and the ability to work with large datasets.
• Proficiency in Python and/or Java/Scala.
• Understanding of data modeling, partitioning, and common data/file formats.
• Experience supporting production systems and resolving issues quickly and effectively.
• Familiarity with DevOps practices, Git, automated deployments, and CI/CD workflows.

Preferred Skills

• Experience with Looker and/or LookML.
• Knowledge of GCS, BigLake, Hive, Apache Iceberg, and Parquet.
• Exposure to Terraform or other Infrastructure-as-Code tools.
• Experience building Kafka-based streaming solutions.
• Background in migrating workloads from on-premises Hadoop/Hive environments to GCP.
• Experience supporting data lake or platform modernization initiatives.
• Familiarity with data quality, governance, lineage, and metadata management.

Horizontal is committed to fostering an inclusive, respectful, and equitable environment where people from all backgrounds can thrive. We value diverse perspectives and encourage candidates to apply even if they do not meet every preferred qualification.

For those that join the team, we offer competitive compensation and benefits including medical, dental, vision, and retirement.

Applications will be accepted for 4 weeks. The pay range for this role is $37 - $71 per hour based on qualifications and experience.