Cloud Data Engineer - Spark, Scala, Airflow | Hybrid Madrid

Inetum; Madrid; Hybrid; EUR 60,000 - 90,000; Full time

Job summary

Inetum in Madrid is seeking a senior Data Engineer to migrate Hadoop infrastructure to cloud-based Kubernetes, Spark on Scala, and Airflow as a service. You will design data transformations, ensure data quality, and collaborate with cross-functional teams to deliver scalable data solutions.

The role requires strong SQL/NoSQL skills, knowledge of HDFS, S3/COS, Parquet/ORC formats, and experience with CI/CD pipelines, testing, and documentation.

Qualifications

• Strong knowledge of Spark on Scala.
• Proficient in Hadoop, HDFS and SQL/NoSQL databases.
• Experience with cloud data services (COS/S3) and Parquet/ORC formats.
• Shell scripting and SDLC awareness; Agile experience.
• Ability to design data transformations and data quality processes.

Responsibilities

• Migrate the Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, COS, Spark as a service, and Airflow as a service.
• Implement data transformation and quality to ensure data consistency and accuracy using Spark/Scala and SQL.
• Set up CI/CD pipelines to automate deployments, unit testing and development management.
• Write and conduct unit and validation tests to ensure accuracy and integrity of code developed.
• Automate data pipelines and streamline data ingestion with orchestrators (Airflow as a Service).
• Write technical documentation to ensure knowledge capitalization.
• Collaborate with cross-functional teams to understand data requirements and deliver solutions.

Skills

Spark

Scala

SQL

NoSQL

Hadoop

CI/CD

Agile

SDLC

Tools

Dremio

Elasticsearch

Kibana

HVault

Job description

Inetum in Madrid is seeking a senior Data Engineer to migrate Hadoop infrastructure to cloud-based Kubernetes, Spark on Scala, and Airflow as a service. You will design data transformations, ensure data quality, and collaborate with cross-functional teams to deliver scalable data solutions.

The role requires strong SQL/NoSQL skills, knowledge of HDFS, S3/COS, Parquet/ORC formats, and experience with CI/CD pipelines, testing, and documentation.