Senior Azure Data Engineer: PySpark, DataOps & DevOps Lead

Arbuthnot Latham; Greater London; Hybrid; GBP 36,000 - 76,000; Full time

Benefits offered by this job

One day a week from home

Pension

Private healthcare

Life assurance 4x

Discretionary bonus

Cycle to Work

Gym Scheme

Health Assessment

Season Ticket/Travel loans

Dental insurance

Job summary

Arbuthnot Latham in London seeks a data engineer to develop and maintain complex data systems supporting operations and reporting. You will build ingest, transformation, analytics and data publishing pipelines to meet business requirements.

Join a team promoting Data Engineering and DevOps best practices, ensuring code quality, security and compliance. You will mentor junior engineers and collaborate with business stakeholders to deliver end-to-end data solutions.

Qualifications

• Significant hands-on experience with the Azure Data Stack, especially ADF and Synapse; experience with Microsoft Fabric is a plus
• Strong Python and PySpark experience; data pipeline development
• DevOps and DataOps experience with engineering, testing, and service delivery
• Understanding of Infrastructure as Code concepts; Terraform experience a plus
• Demonstrable experience of data pipeline testing, including automated testing, data validation, and code assurance
• Experience working in Agile delivery projects
• Understanding of data formats, data security, GDPR, data privacy, and information security
• Strong communication with technical and non-technical communities
• Experience mentoring less-experienced developers
• Industry-focused degree or equivalent working experience
• Azure certifications desirable

Responsibilities

• Develop and maintain complex data systems that support operations and reporting needs
• Build data ingest, transformation, analytics, and data publishing pipelines to meet business requirements
• Support the live platform day to day, resolving issues and handling business requests within our SLAs
• Enable and promote Data Engineering best practices and DevOps standards, ensuring good code quality and compliance with relevant standards
• Provide mentorship to junior engineers and help them enhance their skills and knowledge
• Promote a collaborative environment to deliver and integrate end-to-end data solutions
• Identify and address technical debt, continuously improving data processes and workflows
• Work collaboratively with business stakeholders and operational teams to resolve issues and minimise the defect backlog
• Support estimation processes to aid in planning and portfolio management
• Keep abreast of industry technical best practices and new trends to help meet current and future organisational needs
• Manage risks inherent to the role by observing internal policies and procedures

Skills

Azure Data Stack

Python

PySpark

DevOps

DataOps

Agile delivery

Data security & GDPR

Mentoring

Communication

Education

Industry-focused degree

Tools

Azure

Fabric

Terraform

Python

PySpark

MLOps

Job description

Arbuthnot Latham in London seeks a data engineer to develop and maintain complex data systems supporting operations and reporting. You will build ingest, transformation, analytics and data publishing pipelines to meet business requirements.

Join a team promoting Data Engineering and DevOps best practices, ensuring code quality, security and compliance. You will mentor junior engineers and collaborate with business stakeholders to deliver end-to-end data solutions.