Senior Data Engineer Apache Spark, Python, Azure, SQL

DCG; United Kingdom; A distancia; GBP 80.000 - 120.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Private medical care

Co-financing for the sports card

Dedicated consultant support

Descripción de la vacante

DCG, a recruitment company, seeks a data engineer to design, develop, and maintain scalable data solutions across Azure Synapse, Azure Databricks, and Data Lakehouse environments.

You will implement ETL/ELT processes, build Databricks workflows, and collaborate with architects and stakeholders to translate business needs into technical solutions.

The role requires strong Spark, Python, SQL skills, experience with Unity Catalog, Delta Live Tables, CI/CD, and a proactive approach.

Formación

• Strong experience with Apache Spark, Python, and SQL.
• Proven experience in building and maintaining ETL/ELT processes.
• Practical experience with Databricks, Delta Live Tables (DLT), and Unity Catalog.
• Understanding of Lakehouse architecture, data governance, security, compliance, and operational excellence.
• Familiarity with CI/CD, Azure DevOps, infrastructure automation, monitoring, and logging.
• Strong analytical and problem-solving skills.
• Ability to work independently with a high level of ownership and accountability.
• Proactive approach to identifying risks, blockers, and improvement opportunities.
• Strong communication and stakeholder management skills across technical and business teams.
• Experience working in dynamic environments with frequently changing priorities and cross-functional teams.
• Ability to make pragmatic decisions and deliver results despite incomplete requirements.
• Adaptability, resilience, self-organization, and ability to independently drive initiatives and propose practical solutions.
• English proficiency at minimum B2+ level

Responsabilidades

• Design, develop, and maintain scalable data solutions across Azure Synapse, Azure Databricks, and Data Lakehouse environments
• Implement and optimize ETL/ELT processes for ingesting, transforming, and serving data across multiple business domains
• Develop and maintain data processing solutions using Python, SQL, and Apache Spark
• Support migration and modernization initiatives, including Unity Catalog adoption, platform upgrades, and data platform transformations
• Build and maintain Databricks workflows, Delta Live Tables (DLT), metadata-driven frameworks, and orchestration processes
• Collaborate with architects, analysts, and business stakeholders to translate business requirements into technical solutions
• Ensure data quality, reliability, performance, and security across the data platform
• Troubleshoot production incidents, perform root cause analysis, and implement corrective actions
• Contribute to CI/CD, infrastructure automation, monitoring, logging, and operational best practices
• Optimize data pipelines, storage structures, and processing workloads for efficiency and cost reduction
• Participate in solution and architecture reviews while promoting engineering standards and governance principles
• Support knowledge sharing, mentoring, and technical development within the team
• Collaborate with Data Architects and platform teams to align solutions with enterprise data architecture, Lakehouse principles, and governance requirements
• Support reporting, analytics, and AI/ML initiatives by providing reliable and scalable data foundations

Conocimientos

Apache Spark

Python

SQL

Databricks

Delta Live Tables

Unity Catalog

CI/CD

Herramientas

Azure Synapse

Azure Databricks

Data Lakehouse

Descripción del empleo

As a recruitment company, DCG understands that every business is powered by experienced professionals. Our management style and partnership approach enable us to meet your needs and provide continuous support. Due to our ongoing growth and the large number of recruitment projects we undertake for our partners, we are currently looking for:

Responsibilities

• Design, develop, and maintain scalable data solutions across Azure Synapse, Azure Databricks, and Data Lakehouse environments


• Implement and optimize ETL/ELT processes for ingesting, transforming, and serving data across multiple business domains


• Develop and maintain data processing solutions using Python, SQL, and Apache Spark


• Support migration and modernization initiatives, including Unity Catalog adoption, platform upgrades, and data platform transformations


• Build and maintain Databricks workflows, Delta Live Tables (DLT), metadata-driven frameworks, and orchestration processes


• Collaborate with architects, analysts, and business stakeholders to translate business requirements into technical solutions


• Ensure data quality, reliability, performance, and security across the data platform


• Troubleshoot production incidents, perform root cause analysis, and implement corrective actions


• Contribute to CI/CD, infrastructure automation, monitoring, logging, and operational best practices


• Optimize data pipelines, storage structures, and processing workloads for efficiency and cost reduction


• Participate in solution and architecture reviews while promoting engineering standards and governance principles


• Support knowledge sharing, mentoring, and technical development within the team


• Collaborate with Data Architects and platform teams to align solutions with enterprise data architecture, Lakehouse principles, and governance requirements


• Support reporting, analytics, and AI/ML initiatives by providing reliable and scalable data foundations

Requirements

• Strong experience with Apache Spark, Python, and SQL


• Proven experience in building and maintaining ETL/ELT processes


• Practical experience with Databricks, Delta Live Tables (DLT), and Unity Catalog


• Understanding of Lakehouse architecture, data governance, security, compliance, and operational excellence


• Familiarity with CI/CD, Azure DevOps, infrastructure automation, monitoring, and logging


• Strong analytical and problem-solving skills


• Ability to work independently with a high level of ownership and accountability


• Proactive approach to identifying risks, blockers, and improvement opportunities


• Strong communication and stakeholder management skills across technical and business teams


• Experience working in dynamic environments with frequently changing priorities and cross-functional teams


• Ability to make pragmatic decisions and deliver results despite incomplete requirements


• Adaptability, resilience, self-organization, and ability to independently drive initiatives and propose practical solutions


• English proficiency at minimum B2+ level

Offer

• Private medical care


• Co-financing for the sports card


• Constant support of dedicated consultant

Personal Consulting Agency (License No. 4642)