Senior Data Engineer

Hashlist; Auburn Hills (MI); On-site; USD 120,000 - 180,000; Full time

Job summary

Hashlist is seeking a Senior Data Engineer to design, build, and optimise robust data pipelines for batch and real-time processing at scale. You will work at the intersection of software engineering and data science to ensure scalable, reliable data architectures powering B2B data products.

You will design complex Spark-based pipelines, manage cloud infrastructure with Terraform, and implement CI/CD workflows with GitHub Actions, while upholding rigorous testing and code quality standards.

Qualifications

• 5+ years in data engineering and the software development lifecycle.
• 4+ years hands-on with AWS cloud services.
• 4+ years building production data applications across relational and columnar data stores.
• Deep expertise in Apache Spark - Core, SQL, and Structured Streaming.
• Strong proficiency in Scala or Java for JVM-based production development.
• Advanced SQL for transformation, analysis, and performance tuning.
• Hands-on experience with Terraform and GitHub Actions.
• Experience with Apache Flink for low-latency stream processing.
• Proficiency in Python for automation and scripting.
• Familiarity with Kafka, event-driven architectures, and streaming data.
• Experience with workflow engines — Airflow, Luigi, or Azure Data Factory.
• Familiarity with Lakehouse architectures (Delta Lake, Iceberg) or NoSQL databases.
• Degree in Computer Science, Engineering, Mathematics, or related field preferred.

Responsibilities

• Design and implement complex data processing pipelines using Apache Spark
• Build scalable, distributed systems for high-throughput data streams and large-scale batch processing
• Design and implement real-time pipelines with strong data quality and validation controls
• Manage and provision cloud infrastructure using Terraform
• Implement CI/CD workflows using GitHub Actions for automated testing and deployment
• Uphold rigorous engineering standards - unit/integration testing, code reviews, and clean documentation
• Translate business requirements into technical specifications alongside cross-functional stakeholders

Skills

Data engineering

AWS

Spark

Scala/Java

SQL

Python

Kafka

Airflow

Terraform

GitHub Actions

Flink

Lakehouse

NoSQL

Education

Bachelor's degree in CS/Engineering/Math

Tools

Terraform

GitHub Actions

Delta Lake

Iceberg

NoSQL databases

Azure Data Factory

Airflow

Job description

We're looking for a Senior Data Engineer to design, build, and optimise robust data pipelines that process massive datasets in both batch and real-time. Working at the intersection of software engineering and data science, you'll ensure our data architecture is scalable, reliable, and follows industry best practices - building systems that power B2B data products at global scale.

• Location: Auburn Hills
• Contract type: Permanent/Contract

Responsibilities:

• Design and implement complex data processing pipelines using Apache Spark
• Build scalable, distributed systems for high-throughput data streams and large-scale batch processing
• Design and implement real-time pipelines with strong data quality and validation controls
• Manage and provision cloud infrastructure using Terraform
• Implement CI/CD workflows using GitHub Actions for automated testing and deployment
• Uphold rigorous engineering standards - unit/integration testing, code reviews, and clean documentation
• Translate business requirements into technical specifications alongside cross-functional stakeholders

Qualifications:

• 5+ years in data engineering and the software development lifecycle
• 4+ years hands‑on with AWS cloud services
• 4+ years building production data applications across relational and columnar data stores
• Deep expertise in Apache Spark - Core, SQL, and Structured Streaming
• Strong proficiency in Scala or Java for JVM‑based production development
• Advanced SQL for transformation, analysis, and performance tuning
• Hands‑on experience with Terraform and GitHub Actions
• Experience with Apache Flink for low‑latency stream processing
• Proficiency in Python for automation and scripting
• Familiarity with Kafka, event‑driven architectures, and streaming data
• Experience with workflow engines — Airflow, Luigi, or Azure Data Factory
• Familiarity with Lakehouse architectures (Delta Lake, Iceberg) or NoSQL databases
• Degree in Computer Science, Engineering, Mathematics, or related field preferred