Senior Software Data Engineer

HG Insights; Paris; Sur place; EUR 60 000 - 80 000; Plein temps

Résumé du poste

A leading technology intelligence provider seeks a Mid-Senior level engineer to develop and optimize their Big Data Insights Platform. The ideal candidate has 5+ years of experience in big data systems, expertise in Spark development, and a proven track record in cloud services. Join the team in Paris to solve complex data challenges and lead the future of B2B intelligence at scale.

Qualifications

• 5+ years of experience in building production-grade big data systems.
• Expertise in optimization of batch/streaming jobs.
• Leadership experience as a technical lead.

Responsabilités

• Lead the development of the Big Data Insights Platform.
• Mentor engineers and establish best practices.
• Build and troubleshoot complex data pipeline issues.
• Collaborate in agile workflows.
• Ensure security and compliance across data workflows.

Connaissances

Scala/Java for Spark development

Cloud platforms (AWS EC2, S3, EMR)

CI/CD automation

Agile practices

Data pipeline troubleshooting

Machine learning pipelines

Formation

BS/MS/Ph.D. in Computer Science or related field

Outils

Databricks

Hadoop/Spark ecosystems

Elasticsearch

Terraform

Docker

Kubernetes

Description du poste

HG Insights is the global leader in technology intelligence, delivering actionable AI‑driven insights through advanced data science and scalable big data solutions. Our Big Data Insights Platform processes billions of unstructured documents and powers a vast data lake, enabling enterprises to make strategic, data‑driven decisions. Join our team to solve complex data challenges at scale and shape the future of B2B intelligence.

What You’ll Do:

• Design, build, and optimize large‑scale distributed data pipelines for processing billions of unstructured documents using Databricks, Apache Spark, and cloud‑native big data tools.
• Architect and scale enterprise‑grade big‑data systems, including data lakes, ETL/ELT workflows, and syndication platforms for customer‑facing Insights‑as‑a‑Service (InaaS) products.
• Collaborate with product teams to develop features across databases, backend services, and frontend UIs that expose actionable intelligence from complex datasets.
• Implement cutting‑edge solutions for data ingestion, transformation, and analytics using Hadoop/Spark ecosystems, Elasticsearch, and cloud services (AWS EC2, S3, EMR).
• Drive system reliability through automation, CI/CD pipelines (Docker, Kubernetes, Terraform), and infrastructure‑as‑code practices.

What You’ll Be Responsible For

• Leading the development of our Big Data Insights Platform, ensuring scalability, performance, and cost‑efficiency across distributed systems.
• Mentoring engineers, conducting code reviews, and establishing best practices for Spark optimization, data modeling, and cluster resource management.
• Building & troubleshooting complex data pipeline issues, including performance tuning of Spark jobs, query optimization, and data quality enforcement.
• Collaborating in agile workflows (daily stand‑ups, sprint planning) to deliver features rapidly while maintaining system stability.
• Ensuring security and compliance across data workflows, including access controls, encryption, and governance policies.

What You’ll Need

• BS/MS/Ph.D. in Computer Science or related field, with 5+ years of experience building production‑grade big data systems.
• Expertise in Scala/Java for Spark development, including optimization of batch/streaming jobs and debugging distributed workflows.
• Proven track record with Databricks, Hadoop/Spark ecosystems, and SQL/NoSQL databases (MySQL, Elasticsearch).
• Cloud platforms (AWS EC2, S3, EMR) and infrastructure‑as‑code tools (Terraform, Kubernetes).
• RESTful APIs, microservices architectures, and CI/CD automation.
• Leadership experience as a technical lead, including mentoring engineers and driving architectural decisions.
• Strong understanding of agile practices, distributed computing principles, and data lake architectures.
• Airflow orchestration (DAGs, operators, sensors) and integration with Spark/Databricks.
• 7+ years of designing, modeling and building big data pipelines in an enterprise work setting.

Nice‑to‑Haves

• Experience with machine learning pipelines (Spark MLlib, Databricks ML) for predictive analytics.
• Knowledge of data governance frameworks and compliance standards (GDPR, CCPA).
• Contributions to open‑source big data projects or published technical blogs/papers.
• DevOps proficiency in monitoring tools (Prometheus, Grafana) and serverless architectures.

Seniority level

• Mid‑Senior level

Employment type

• Full‑time

Job function

• Information Technology

Industries

• Software Development

Lille, Hauts‑de‑France, France