Data Engineer: Spark Pipelines & AI-Driven Data Tools

Moody's Corporation; Brussel Hoofdstad; Sur place; EUR 70 000 - 110 000; Plein temps

Résumé du poste

Moody’s Corporation is seeking a data engineering candidate to design, develop, and optimize scalable data pipelines. You will work with Spark-based processes, Delta Lake, Parquet, and graph analytics in a cloud environment (AWS preferred).

Strong SQL and testing practices are required, with collaboration across Product, Data Science, and Engineering teams. The role emphasizes building robust ETL workflows, data quality checks, and documentation while exploring AI-assisted development tools to

Qualifications

• Experience building scalable data applications with Spark and Scala preferred; Python acceptable.
• Hands-on Databricks experience in a cloud environment, AWS preferred.
• Strong understanding of distributed computing, data partitioning, and performance optimisation.
• Experience with Parquet, Delta Lake, and JSON data formats.
• Strong SQL skills for data transformation and querying.
• Knowledge of graph data structures, clustering, and similarity approaches.
• Familiarity with TF-IDF and similarity metrics.
• Proficiency with Git, Gradle, Maven, or SBT for builds.
• Awareness of code quality, testing (ScalaTest/JUnit), and testing approaches.
• Ability to use AI-assisted development tools and collaboration in cross-functional teams.

Responsabilités

• Support design, development, and optimisation of data pipelines in a scalable data engineering environment.
• Develop Spark-based data pipelines for large-scale data processing.
• Implement parsing, normalisation, and standardisation logic across datasets.
• Assist in graph-based analytics and entity resolution workflows.
• Build and maintain ETL workflows ingesting Delta Lake, Parquet, and S3 data.
• Contribute to data quality checks and incremental processing approaches.
• Write and maintain unit and integration tests to support reliability.
• Support monitoring and troubleshooting of data pipelines.
• Collaborate with Product, Data Science, and Engineering teams.
• Document data flows and technical solutions; participate in code reviews and best practices.

Connaissances

Spark

Scala

Python

Databricks

AWS

SQL

Graph analytics

TF-IDF

Testing

AI-assisted tools

Communication

Collaboration

Learning new tech

Formation

Bachelor’s or Master’s in CS/IT

Equivalent practical experience

Outils

Apache Spark

Databricks

Git

Gradle

Maven

SBT

ScalaTest

JUnit

Description du poste

Moody’s Corporation is seeking a data engineering candidate to design, develop, and optimize scalable data pipelines. You will work with Spark-based processes, Delta Lake, Parquet, and graph analytics in a cloud environment (AWS preferred).

Strong SQL and testing practices are required, with collaboration across Product, Data Science, and Engineering teams. The role emphasizes building robust ETL workflows, data quality checks, and documentation while exploring AI-assisted development tools to