Data engineer in AI

Artefact; Greater London; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

Artefact is a data service provider helping organisations transform data into business impact. We seek a senior data engineer lead to design and maintain robust data pipelines using SQL and Python, with Databricks, Snowflake, Airflow and PySpark in a hybrid London-area role.

You will lead a team of data engineers, implement CI/CD for pipelines, ensure data governance and security, and collaborate with architects and analysts to meet business needs while exploring AI-driven improvements.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 3+ years of industry experience in data engineering with strong proficiency in SQL, Python, and big data technologies.
• Expertise in building scalable data workflows using cloud-native orchestration tools and distributed data processing frameworks.
• Solid understanding of CI/CD principles and DevOps/DevSecOps practices.
• Experience leading on project workstreams.
• Excellent problem-solving skills and the ability to work with ambiguity.
• Proficient in leveraging AI-assisted workflows to optimise task efficiency.

Responsibilities

• Design, build, and maintain scalable data pipelines using SQL and Python, leveraging Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and PySpark.
• Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms.
• Implement CI/CD practices for data pipelines to improve efficiency and quality of data processing.
• Work closely with data architects, analysts, and stakeholders to translate requirements into technical implementations.
• Oversee and mentor a team of data engineers to ensure high-quality project deliverables.
• Advise on data governance, security, and compliance within the organisation.
• Continuously evaluate new technologies and AI trends to enhance data engineering capabilities.

Skills

SQL

Python

Big data

CI/CD

DevOps/DevSecOps

Data governance

Leadership

Education

Bachelor's or Master’s in Computer Science or related field

Tools

Databricks

Snowflake

Azure Data Factory

AWS Glue

Apache Airflow

PySpark

Job description

Описание:

Artefact is a new generation of data service provider, specialising in data consulting and data-driven digital marketing, dedicated to transforming data into business impact across the entire value chain of organisations. We are proud to say we’re enjoying skyrocketing growth.

Задачи:

• Design, build, and maintain scalable and robust data pipelines using SQL and Python, and leveraging technologies such as: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark.
• Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms.
• Implement continuous integration and continuous deployment (CI/CD) practices for data pipelines to improve efficiency and quality of data processing.
• Work closely with data architects, analysts, and other stakeholders to understand business requirements and translate them into technical implementations.
• Oversee and manage a team of data engineers, providing guidance and mentorship to ensure high-quality project deliverables.
• Knowledge of best practices in data governance, security, and compliance within the organisation.
• Continuously evaluate new technologies, AI trends and tools to enhance the capabilities of the data engineering function.

Требования:

• Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.
• 3+ years of industry experience in data engineering with a strong technical proficiency in SQL, Python, and big data technologies.
• Expertise in building scalable data workflows using cloud-native orchestration tools and distributed data processing frameworks.
• Solid understanding of CI/CD principles and DevOps/DevSecOps practices.
• Experience leading on project workstreams.
• Excellent problem-solving skills and the ability to work with ambiguity.
• Proficient in leveraging AI-assisted workflows to optimise task efficiency.
• Strong communication and interpersonal skills.

Будет плюсом:

• Certifications in Azure, AWS, or GCP;
• Certifications in Databricks, Snowflake or similar technologies;
• Experience in the leading large scale data engineering projects;
• Excellent understanding of data architecture involving data mesh, data lake, data warehouse and data lakehouse;
• Demonstrated experience with Infrastructure as Code tooling such as Terraform

Условия:

Join one of the fastest-growing global data consulting companies and help leading organisations transform through data and AI. Work on cutting-edge projects with modern technologies including AI, Machine Learning, Cloud, and DevOps, while continuously developing your technical, consulting, and leadership skills. Be part of a diverse, international, and action-oriented team of experts across data, consulting, engineering, and digital marketing. We value initiative, innovation, and ownership. Your ideas matter, and you’ll have the opportunity to make a real impact from day one. Hybrid work arrangement: two days per week working from the office