Senior Data Engineer: Real-Time Pipelines & Spark

Hashlist; Poissy; Sur place; EUR 85 000 - 125 000; Plein temps

Résumé du poste

Hashlist is seeking a Senior Data Engineer to design, build, and optimise robust data pipelines handling massive datasets in batch and real-time, ensuring scalable and reliable data architecture for B2B products at global scale.

You will work at the intersection of software engineering and data science, applying best practices, CI/CD with GitHub Actions, Terraform for cloud, and Spark/Scala/SQL to deliver high-quality data products across the organization in Poissy, Île-de-France.

Qualifications

• Degree in Computer Science, Engineering, Mathematics, or related field preferred.
• 5+ years in data engineering and the software development lifecycle.
• 4+ years hands-on with AWS cloud services.
• Deep expertise in Apache Spark — Core, SQL, and Structured Streaming.

Responsabilités

• Design and implement complex data processing pipelines using Apache Spark.
• Build scalable, distributed systems for high-throughput data streams and large-scale batch processing.
• Design and implement real-time pipelines with strong data quality and validation controls.
• Manage and provision cloud infrastructure using Terraform.
• Implement CI/CD workflows using GitHub Actions for automated testing and deployment.
• Uphold rigorous engineering standards — unit/integration testing, code reviews, and clean documentation.
• Translate business requirements into technical specifications alongside cross-functional stakeholders.

Connaissances

Apache Spark

Scala or Java

SQL

Python

Terraform

GitHub Actions

Kafka

Airflow

Azure Data Factory

NoSQL databases

Formation

Degree in Computer Science, Engineering, Mathematics

Outils

Terraform

GitHub Actions

Airflow

Luigi

Azure Data Factory

Description du poste

Hashlist is seeking a Senior Data Engineer to design, build, and optimise robust data pipelines handling massive datasets in batch and real-time, ensuring scalable and reliable data architecture for B2B products at global scale.

You will work at the intersection of software engineering and data science, applying best practices, CI/CD with GitHub Actions, Terraform for cloud, and Spark/Scala/SQL to deliver high-quality data products across the organization in Poissy, Île-de-France.