Lead Data Engineer - AI Pipelines & Cloud

Artefact; Greater London; Hybrid; GBP 85,000 - 120,000; Full time

Benefits offered by this job

Hybrid work model

Job summary

Artefact seeks a Senior Data Engineer to lead data projects with Python/SQL pipelines, Databricks, Snowflake, and cloud tools. You will mentor engineers, implement CI/CD, and drive governance, security, and performance across platforms.

You will collaborate with architects, analysts, and stakeholders to translate business needs into robust data solutions. This role offers a hybrid work model with growth across data, AI, and cloud capabilities.

Qualifications

• 4+ years of industry experience in data engineering with strong SQL, Python, and big data tech.
• Experience with cloud-native orchestration tools and distributed processing frameworks.
• Experience with Infrastructure as Code tooling such as Terraform.
• Knowledge of CI/CD principles and DevOps/DevSecOps practices.
• Proven leadership skills and experience managing data engineering teams.
• Excellent problem-solving and ability to work with ambiguity.
• Proficient in AI-assisted workflows to optimise task efficiency.
• Strong communication and interpersonal skills.
• Understanding of data mesh, lake, warehouse, and lakehouse architectures.

Responsibilities

• Design, build, and maintain scalable data pipelines using SQL and Python with Databricks, Snowflake, and cloud tools.
• Lead integration of complex data systems ensuring data consistency across platforms.
• Implement CI/CD practices for data pipelines to improve efficiency and quality.
• Collaborate with data architects and analysts to translate business needs into implementations.
• Oversee and mentor a team of data engineers to deliver high-quality projects.
• Develop and enforce data governance, security, and compliance practices.
• Optimize data retrieval and develop dashboards and reports for business teams.
• Evaluate new technologies to enhance the data engineering function.

Skills

SQL

Python

Databricks

Snowflake

Azure Data Factory

AWS Glue

Apache Airflow

PySpark

CI/CD

DevOps

Leadership

Data Mesh / Data Lake / Data Warehouse

Education

Bachelor's or Master's degree in Computer Science, Engineering, or related field

Tools

Terraform

Job description

Artefact seeks a Senior Data Engineer to lead data projects with Python/SQL pipelines, Databricks, Snowflake, and cloud tools. You will mentor engineers, implement CI/CD, and drive governance, security, and performance across platforms.

You will collaborate with architects, analysts, and stakeholders to translate business needs into robust data solutions. This role offers a hybrid work model with growth across data, AI, and cloud capabilities.