Azure Data Engineer

QuickOps Consulting; Portugal; Presencial; EUR 60 000 - 90 000; Tempo integral

Resumo da oferta

QuickOps Consulting seeks a data engineer to design and scale data platforms using Databricks and PySpark, with strong Python and SQL skills. You will build high-volume batch and streaming pipelines and optimize Spark workloads.

Responsibilities include implementing Delta Lake and Medallion Architecture, enabling secure data governance with Unity Catalog, and integrating with cloud storage such as Azure ADLS Gen2 or AWS S3.

Qualificações

• Strong hands-on experience with Databricks, PySpark, Python and SQL.
• Deep understanding of Spark architecture, including partitions, memory management, shuffle operations and performance tuning.
• Expert-level SQL skills, including complex JOINs, Window Functions, CTEs, aggregations and query tuning.
• Experience with data modelling, ETL/ELT, Delta Lake, Medallion Architecture and batch/real-time data pipelines.
• Experience with Azure and Databricks integration, ideally with Azure ADLS Gen2, Snowflake or AWS S3.
• Experience with Git and CI/CD; knowledge of Terraform, Databricks Asset Bundles, pytest or data validation tools such as Great Expectations is a plus.

Responsabilidades

• Design, build and maintain scalable data platforms and high-volume batch and streaming pipelines using Databricks and PySpark
• Develop data processing workflows using Spark, Apache Beam and Databricks Workflows
• Implement data architectures using Delta Lake, Medallion Architecture and incremental loading patterns such as CDC, MERGE INTO and Auto Loader
• Ensure data accuracy, security and governance through appropriate access controls, encryption and Databricks Unity Catalog
• Collaborate with cross-functional teams to deliver reliable data solutions and extract data from databases, file systems and APIs
• Monitor and optimize data platforms and Spark applications for performance, scalability and high availability.

Conhecimentos

Databricks

PySpark

Python

SQL

Spark architecture

Delta Lake

Medallion Architecture

Auto Loader

Git & CI/CD

Great Expectations

Ferramentas

Databricks

Delta Lake

Apache Beam

Terraform

Great Expectations

Azure ADLS Gen2

Snowflake

AWS S3

Descrição da oferta de emprego

The ideal candidate will have strong experience in data engineering, with a solid background in Databricks, PySpark, Python and SQL. They should be experienced in designing scalable data platforms and pipelines, working with large datasets, optimizing Spark workloads and implementing secure, reliable data solutions.

Responsibilities

• Design, build and maintain scalable data platforms and high-volume batch and streaming pipelines using Databricks and PySpark
• Develop data processing workflows using Spark, Apache Beam and Databricks Workflows
• Implement data architectures using Delta Lake, Medallion Architecture and incremental loading patterns such as CDC, MERGE INTO and Auto Loader
• Ensure data accuracy, security and governance through appropriate access controls, encryption and Databricks Unity Catalog
• Collaborate with cross-functional teams to deliver reliable data solutions and extract data from databases, file systems and APIs
• Monitor and optimize data platforms and Spark applications for performance, scalability and high availability.

Qualifications

• Strong hands-on experience with Databricks, PySpark, Python and SQL
• Deep understanding of Spark architecture, including partitions, memory management, shuffle operations and performance tuning
• Expert-level SQL skills, including complex JOINs, Window Functions, CTEs, aggregations and query tuning
• Experience with data modelling, ETL/ELT, Delta Lake, Medallion Architecture and batch/real-time data pipelines
• Experience with Azure and Databricks integration, ideally with Azure ADLS Gen2, Snowflake or AWS S3
• Experience with Git and CI/CD; knowledge of Terraform, Databricks Asset Bundles, pytest or data validation tools such as Great Expectations is a plus.