Data Engineer

Krystal Clarity; Greater London; Hybrid; GBP 45,000 - 65,000; Full time

Benefits offered by this job

Mentorship from senior engineers

Remote work flexibility

Diverse client projects

Job summary

Krystal Clarity is seeking a Mid Level Data Engineer with 2–4 years of experience to design, build, and deploy data pipelines on Azure Databricks. You will ingest data from REST APIs and collaborate with senior engineers on real client projects, using Python and SQL to create scalable solutions.

You will work with AI coding tools to accelerate development, contribute to CI/CD and infrastructure as code, and engage clients directly to gather requirements and present progress.

Qualifications

• Bachelor's or Master's in a STEM subject (Computer Science, Maths, Physics, Engineering, ...)
• 2 to 4 years of experience as a data engineer working on Azure Databricks or in the Azure/AWS ecosystem.
• Hands-on experience with Python (PySpark strongly preferred) and strong SQL skills is required.
• Hands-on experience using AI coding tools (Claude Code, Cursor, Codex or similar) in real development workflows, with judgement on when and how to apply them.
• Experience with infrastructure-as-code, ideally Terraform, is required.
• Familiarity with CI/CD workflows and tools like GitHub Actions or Azure Pipelines.
• Working knowledge of Unity Catalog and data governance fundamentals.
• Experience ingesting data from REST APIs and various third-party systems.
• Experience with the wider Azure data ecosystem (Data Factory, Storage, Event Hub) is desirable.
• Expertise in other relevant technologies is desirable: Lakeflow Connect, Declarative Pipelines, Databricks Apps, Vector Search or MLOps, dbt, Snowflake, Debezium/Kafka/Streaming, Power BI.

Responsibilities

• Design, build, and optimise scalable data pipelines using Databricks (PySpark, SQL, Delta Lake) on Azure.
• Build ingestion from REST APIs, including incremental and near-real-time load patterns.
• Collaborate with senior engineers to understand requirements and implement solutions for clients.
• Work directly with client stakeholders, including face-to-face: gathering requirements, presenting solutions and communicating progress to technical and non-technical audiences.
• Write clean, maintainable code in Python, applying best practices for testing and code quality.
• Use AI coding assistants day to day to accelerate development, testing and documentation, while maintaining code quality.
• Contribute to CI/CD workflows using GitHub Actions and Azure Pipelines, including automation of infrastructure and deployments.
• Use Terraform and Declarative Automation Bundles to manage infrastructure as code, alongside scripting tools (Bash, PowerShell).
• Apply strong data governance and quality practices, including Unity Catalog, RBAC and PII handling, to ensure data integrity.
• Document solutions, pipelines, and design decisions for future maintainability.
• Continuously learn and explore new tools, frameworks, and best practices in data engineering.

Skills

Python

SQL

Stakeholder management

Client communication

Team collaboration

Education

Bachelor's or Master's in STEM

Tools

Azure Databricks

Azure

REST APIs

GitHub Actions

Azure Pipelines

Terraform

Unity Catalog

Job description

Krystal Clarity is a rapidly emerging data engineering consultancy and Databricks partner that helps businesses leverage data to achieve their strategic objectives. As a lean, fast-paced, and dynamic company, we are looking for an ambitious Data Engineer who thrives in an entrepreneurial environment and seeks to be part of the foundational team driving the company's growth and success.

Position Overview:

We are seeking a Mid Level Data Engineer with 2 to 4 years of experience in data engineering, with strong exposure to Azure, Databricks and Python. You'll work closely with senior engineers on real-world client projects, designing, building, and deploying data pipelines in the cloud. This role is perfect for someone looking to sharpen their technical skills, work in a collaborative consultancy environment, and take ownership of impactful work.

Responsibilities:

• Design, build, and optimise scalable data pipelines using Databricks (PySpark, SQL, Delta Lake) on Azure.
• Build ingestion from REST APIs, including incremental and near-real-time load patterns, alongside managed connectors such as Lakeflow Connect and Azure Data Factory.
• Collaborate with senior engineers to understand requirements and implement solutions for clients.
• Work directly with client stakeholders, including face-to-face: gathering requirements, presenting solutions and communicating progress to technical and non-technical audiences.
• Write clean, maintainable code in Python, applying best practices for testing and code quality.
• Use AI coding assistants (Claude Code, Cursor, Codex or similar) day to day to accelerate development, testing and documentation, while maintaining code quality.
• Contribute to CI/CD workflows using GitHub Actions and Azure Pipelines, including automation of infrastructure and deployments.
• Use Terraform and Declarative Automation Bundles to manage infrastructure as code, alongside scripting tools (Bash, PowerShell).
• Apply strong data governance and quality practices, including Unity Catalog, RBAC and PII handling, to ensure data integrity.
• Document solutions, pipelines, and design decisions for future maintainability.
• Continuously learn and explore new tools, frameworks, and best practices in data engineering.

Requirements:

• Bachelor's or Master's in a STEM subject (Computer Science, Maths, Physics, Engineering, ...).
• 2 to 4 years of experience as a data engineer working on Azure Databricks or in the Azure/AWS ecosystem.
• Hands-on experience with Python (PySpark strongly preferred) and strong SQL skills is required.
• Hands-on experience using AI coding tools (Claude Code, Cursor, Codex or similar) in real development workflows, with judgement on when and how to apply them.
• Experience with infrastructure-as-code, ideally Terraform, is required.
• Familiarity with CI/CD workflows and tools like GitHub Actions or Azure Pipelines.
• Working knowledge of Unity Catalog and data governance fundamentals.
• Experience ingesting data from REST APIs and various third-party systems.
• Experience with the wider Azure data ecosystem (Data Factory, Storage, Event Hub) is desirable.
• Expertise in other relevant technologies is desirable: Lakeflow Connect, Declarative Pipelines, Databricks Apps, Vector Search or MLOps, dbt, Snowflake, Debezium/Kafka/Streaming, Power BI.
• Ability to work collaboratively in a small, fast-moving team.
• Strong communication and stakeholder-management skills, with the confidence to work with clients face-to-face from day one and the knack for simplifying complex topics.
• Resilience to thrive in a dynamic environment, adeptly managing multiple projects.

What We Offer:

• The opportunity to join a consultancy in its growth stage and directly shape its success.
• Hands-on experience across diverse client projects with mentorship from senior engineers.
• Competitive salary and benefits package.
• Flexibility to work remotely, with occasional in-person collaboration.