Senior Azure Data Engineer: Databricks & Delta Lake

Capco; Greater London; Hybrid; GBP 70,000 - 90,000; Full time

Benefits offered by this job

Discretionary bonus

Competitive pension

Health insurance

Life insurance

Critical illness cover

Continuous training opportunities

Gympass

Holiday flexibility

Job summary

A leading consulting firm is seeking a Senior Azure Data Engineer to design and develop scalable data solutions on the Databricks platform. This role will involve hands-on tasks such as building robust data pipelines, collaborating with clients to address their data challenges, and applying DevOps principles. The ideal candidate will have proficiency in Python, PySpark, and SQL, as well as experience with CI/CD processes. The firm offers a comprehensive benefits package, a hybrid work model, and opportunities for continuous learning.

Qualifications

• Proven hands-on experience with the Databricks platform and orchestration.
• Strong skills in Python, PySpark, and SQL with knowledge of distributed data systems.
• Expertise in developing full lifecycle data pipelines.

Responsibilities

• Design and develop robust pipelines using DeltaLake, Spark Structured Streaming.
• Build event-driven solutions with tools like Kafka and Azure Event Hubs.
• Collaborate with clients to translate data needs into technical solutions.

Skills

Experience with Databricks platform

Strong skills in Python

Proficiency in PySpark

Knowledge of SQL

Expertise in data lakehouse architecture

Tools

Azure DevOps

Kafka

Jenkins

Job description

A leading consulting firm is seeking a Senior Azure Data Engineer to design and develop scalable data solutions on the Databricks platform. This role will involve hands-on tasks such as building robust data pipelines, collaborating with clients to address their data challenges, and applying DevOps principles. The ideal candidate will have proficiency in Python, PySpark, and SQL, as well as experience with CI/CD processes. The firm offers a comprehensive benefits package, a hybrid work model, and opportunities for continuous learning.