Data Engineer GCP Python, Spark, SQL Senior

Keoni; Paris; Sur place; EUR 70 000 - 110 000; Plein temps

Résumé du poste

Demartino Law is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines using Python, Spark and SQL on Google Cloud Platform. You will orchestrate with Airflow, manage storage with BigQuery/Dataproc, and implement infrastructure with Terraform/Terragrunt.

The role requires strong data modeling, governance, and collaboration with product and data teams across Europe. Strong communication and Agile experience are essential.

Qualifications

• At least 3 years of experience as a Data Engineer in a data environment.
• Experience in designing and implementing data pipelines.
• Experience in a Google Cloud Platform (GCP) environment.
• Experience integrating BigQuery and Dataplex Universal Catalog.
• Experience integrating Python and SQL.
• Experience integrating Spark.
• Experience integrating Delta Lake.
• Experience integrating Terraform and Terragrunt.
• Experience integrating GitLab and GitLab CI/CD.
• Experience integrating Agile methodologies: Scrum or Kanban.
• Ability to simplify and communicate purely technical aspects.

Responsabilités

• Collect and centralize various data sources in a scalable way.
• Ensure data compliance and security across pipelines and storage.
• Build and maintain robust, scalable data pipelines.
• Support data exploitation and querying by team members.
• Organize and structure data storage to optimize access and governance.
• Develop and maintain CI/CD workflows for data infra.
• Participate in data engineering meetings and document updates.

Connaissances

Python

Spark

SQL

Airflow

GCP

BigQuery

Dataproc

Terraform

Terragrunt

GitLab CI/CD

Agile

Communication

Outils

Python

Spark

SQL

Airflow

GCP

BigQuery

Dataproc

Pub/Sub

Terraform

Terragrunt

GitLab CI/CD

Description du poste

# Data Engineer GCP Python, Spark, SQL Senior* Data Engineer GCP Python, Spark, SQL SeniorContext: This mission will be carried out in collaboration with the other Data teams, at the heart of the Data platform. This team consists of a product owner, a data steward, and two other data engineers. Our technical stack primarily comprises Python, Spark, and SQL for data processing; Airflow for orchestration; and Cloud Storage and Delta Lake for storage. The stack also includes BigQuery, Dataproc, PubSub, and Cloud Storage services from Google Cloud Platform, as well as Terraform and Terragrunt for infrastructure management.MISSIONS: 1. Collect and centralize various non-homogeneous data sources 2. Ensure data compliance and security 3. Participate in data governance initiatives In interaction with team members, the service consists of: • Actively participating in the various phases of design, planning, and execution of tasks with the team • Building and maintaining robust and scalable data pipelines • Supporting team members in data exploitation and querying • Organizing and structuring data storage • Implementing and maintaining scalable infrastructures capable of meeting the needs and rapid growth of data volume • Building and maintaining CI/CD workflows • Sharing best practices with other data engineers • Ensuring the scalability, security, and stability of environments • Participating in various data engineering meetings • Contributing to and ensuring the updating of documentation . Actively monitoring technological developments in the fieldProfile & Required Expertise: • At least 3 years of experience as a Data Engineer in a data environment • Strong data skills: data structures, code, architecture • Experience in designing and implementing data pipelines • Experience in a Google Cloud Platform (GCP) environment • Experience integrating BigQuery and Dataplex Universal Catalog • Experience integrating Python and SQL • Experience integrating Spark • Experience integrating Delta Lake • Experience integrating Terraform and Terragrunt • Experience integrating GitLab and GitLab CI/CD • Experience integrating Agile methodologies: Scrum or Kanban • Ability to simplify and communicate purely technical aspects Expected Expertise Level: Senior: more than 5 years of significant experience in the type of expertise required**Work location:** Europe