Senior Data Engineer — AI-Driven Data Pipelines

Artefact; Greater London; Hybrid; GBP 70,000 - 100,000; Full time

Benefits offered by this job

Hybrid work model (2 days in office)

Job summary

Artefact, a global data services provider, is seeking a Senior Data Engineer to join its fast-growing team in London. You will design and scale data pipelines, mentor engineers, and align data solutions with business goals.

You will work with SQL, Python and cloud-native tools like Databricks, Snowflake, and Airflow, shaping data governance and CI/CD practices while collaborating with analysts, architects and stakeholders.

Qualifications

• 3+ years of industry experience in data engineering.
• Strong proficiency in SQL and Python.
• Experience with cloud-native orchestration tools and distributed processing frameworks.
• Solid understanding of CI/CD and DevOps/DevSecOps practices.
• Experience leading on project workstreams.
• Excellent problem-solving and communication skills.
• Proficient in leveraging AI-assisted workflows to optimise task efficiency.

Responsibilities

• Design, build, and maintain scalable and robust data pipelines using SQL and Python, and leveraging technologies such as: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark.
• Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms.
• Implement continuous integration and continuous deployment (CI/CD) practices for data pipelines to improve efficiency and quality of data processing.
• Work closely with data architects, analysts, and other stakeholders to understand business requirements and translate them into technical implementations.
• Oversee and manage a team of data engineers, providing guidance and mentorship to ensure high-quality project deliverables.
• Knowledge of best practices in data governance, security, and compliance within the organisation.
• Continuously evaluate new technologies, AI trends and tools to enhance the capabilities of the data engineering function.

Skills

SQL

Python

Databricks

Snowflake

Azure Data Factory

AWS Glue

Apache Airflow

PySpark

CI/CD

DevOps

Leadership

Education

Bachelor's or Master's degree in Computer Science/Engineering

Tools

Databricks

Snowflake

Azure Data Factory

AWS Glue

Apache Airflow

PySpark

Job description

Artefact, a global data services provider, is seeking a Senior Data Engineer to join its fast-growing team in London. You will design and scale data pipelines, mentor engineers, and align data solutions with business goals.

You will work with SQL, Python and cloud-native tools like Databricks, Snowflake, and Airflow, shaping data governance and CI/CD practices while collaborating with analysts, architects and stakeholders.