Hybrid Lead Data Engineer for AI & Pipelines

Artefact; Greater London; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

Artefact is a data service provider helping organisations transform data into business impact. We seek a senior data engineer lead to design and maintain robust data pipelines using SQL and Python, with Databricks, Snowflake, Airflow and PySpark in a hybrid London-area role.

You will lead a team of data engineers, implement CI/CD for pipelines, ensure data governance and security, and collaborate with architects and analysts to meet business needs while exploring AI-driven improvements.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 3+ years of industry experience in data engineering with strong proficiency in SQL, Python, and big data technologies.
• Expertise in building scalable data workflows using cloud-native orchestration tools and distributed data processing frameworks.
• Solid understanding of CI/CD principles and DevOps/DevSecOps practices.
• Experience leading on project workstreams.
• Excellent problem-solving skills and the ability to work with ambiguity.
• Proficient in leveraging AI-assisted workflows to optimise task efficiency.

Responsibilities

• Design, build, and maintain scalable data pipelines using SQL and Python, leveraging Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and PySpark.
• Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms.
• Implement CI/CD practices for data pipelines to improve efficiency and quality of data processing.
• Work closely with data architects, analysts, and stakeholders to translate requirements into technical implementations.
• Oversee and mentor a team of data engineers to ensure high-quality project deliverables.
• Advise on data governance, security, and compliance within the organisation.
• Continuously evaluate new technologies and AI trends to enhance data engineering capabilities.

Skills

SQL

Python

Big data

CI/CD

DevOps/DevSecOps

Data governance

Leadership

Education

Bachelor's or Master’s in Computer Science or related field

Tools

Databricks

Snowflake

Azure Data Factory

AWS Glue

Apache Airflow

PySpark

Job description

Artefact is a data service provider helping organisations transform data into business impact. We seek a senior data engineer lead to design and maintain robust data pipelines using SQL and Python, with Databricks, Snowflake, Airflow and PySpark in a hybrid London-area role.

You will lead a team of data engineers, implement CI/CD for pipelines, ensure data governance and security, and collaborate with architects and analysts to meet business needs while exploring AI-driven improvements.