Lead Data Engineer — Pipelines, AI & Cloud (Hybrid)

twentysix; United Kingdom; Hybrid; GBP 70,000 - 100,000; Full time

Job summary

Artefact is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines using SQL, Python, Databricks, Snowflake and orchestration tools. You will lead a team, collaborate with architects and analysts, and drive data governance and security practices across complex systems.

The role emphasizes CI/CD for data workflows, tooling with cloud-native services, and delivering robust data solutions that enable business insights across platforms.

Qualifications

• Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.
• 3+ years of industry experience in data engineering with a strong technical proficiency in SQL, Python, and big data technologies.
• Expertise in building scalable data workflows using cloud-native orchestration tools and distributed data processing frameworks.
• Demonstrated experience with Infrastructure as Code tooling such as Terraform
• Solid understanding of CI/CD principles and DevOps/DevSecOps practices.
• Proven leadership skills and experience managing data engineering teams.
• Excellent problem-solving skills and the ability to work with ambiguity.
• Proficient in leveraging AI-assisted workflows to optimise task efficiency.
• Strong communication and interpersonal skills.
• Excellent understanding of data architecture involving data mesh, data lake, data warehouse and data lakehouse.

Responsibilities

• Design, build, and maintain scalable and robust data pipelines using SQL and Python, and leveraging Databricks, Snowflake, Azure Data Factory, AWS Glue, Airflow and PySpark.
• Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms.
• Implement CI/CD practices for data pipelines to improve efficiency and quality of data processing.
• Work closely with data architects, analysts, and stakeholders to translate business requirements into technical implementations.
• Oversee and mentor a team of data engineers to ensure high-quality project deliverables.
• Develop and enforce best practices in data governance, security, and compliance.
• Optimise data retrieval and develop dashboards and reports for business teams.
• Continuously evaluate new technologies and tools to enhance data engineering capabilities.

Skills

SQL

Python

Databricks

Snowflake

Azure Data Factory

AWS Glue

Apache Airflow

PySpark

Terraform

CI/CD

DevOps/DevSecOps

Leadership

Education

Bachelor's or Master’s degree in Computer Science, Engineering, or a related field

Tools

Terraform

Job description

Artefact is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines using SQL, Python, Databricks, Snowflake and orchestration tools. You will lead a team, collaborate with architects and analysts, and drive data governance and security practices across complex systems.

The role emphasizes CI/CD for data workflows, tooling with cloud-native services, and delivering robust data solutions that enable business insights across platforms.