Remote Data Engineer – Databricks, Azure & Python

Krystal Clarity; Greater London; Hybrid; GBP 45,000 - 65,000; Full time

Benefits offered by this job

Mentorship from senior engineers

Remote work flexibility

Diverse client projects

Job summary

Krystal Clarity is seeking a Mid Level Data Engineer with 2–4 years of experience to design, build, and deploy data pipelines on Azure Databricks. You will ingest data from REST APIs and collaborate with senior engineers on real client projects, using Python and SQL to create scalable solutions.

You will work with AI coding tools to accelerate development, contribute to CI/CD and infrastructure as code, and engage clients directly to gather requirements and present progress.

Qualifications

• Bachelor's or Master's in a STEM subject (Computer Science, Maths, Physics, Engineering, ...)
• 2 to 4 years of experience as a data engineer working on Azure Databricks or in the Azure/AWS ecosystem.
• Hands-on experience with Python (PySpark strongly preferred) and strong SQL skills is required.
• Hands-on experience using AI coding tools (Claude Code, Cursor, Codex or similar) in real development workflows, with judgement on when and how to apply them.
• Experience with infrastructure-as-code, ideally Terraform, is required.
• Familiarity with CI/CD workflows and tools like GitHub Actions or Azure Pipelines.
• Working knowledge of Unity Catalog and data governance fundamentals.
• Experience ingesting data from REST APIs and various third-party systems.
• Experience with the wider Azure data ecosystem (Data Factory, Storage, Event Hub) is desirable.
• Expertise in other relevant technologies is desirable: Lakeflow Connect, Declarative Pipelines, Databricks Apps, Vector Search or MLOps, dbt, Snowflake, Debezium/Kafka/Streaming, Power BI.

Responsibilities

• Design, build, and optimise scalable data pipelines using Databricks (PySpark, SQL, Delta Lake) on Azure.
• Build ingestion from REST APIs, including incremental and near-real-time load patterns.
• Collaborate with senior engineers to understand requirements and implement solutions for clients.
• Work directly with client stakeholders, including face-to-face: gathering requirements, presenting solutions and communicating progress to technical and non-technical audiences.
• Write clean, maintainable code in Python, applying best practices for testing and code quality.
• Use AI coding assistants day to day to accelerate development, testing and documentation, while maintaining code quality.
• Contribute to CI/CD workflows using GitHub Actions and Azure Pipelines, including automation of infrastructure and deployments.
• Use Terraform and Declarative Automation Bundles to manage infrastructure as code, alongside scripting tools (Bash, PowerShell).
• Apply strong data governance and quality practices, including Unity Catalog, RBAC and PII handling, to ensure data integrity.
• Document solutions, pipelines, and design decisions for future maintainability.
• Continuously learn and explore new tools, frameworks, and best practices in data engineering.

Skills

Python

SQL

Stakeholder management

Client communication

Team collaboration

Education

Bachelor's or Master's in STEM

Tools

Azure Databricks

Azure

REST APIs

GitHub Actions

Azure Pipelines

Terraform

Unity Catalog

Job description

Krystal Clarity is seeking a Mid Level Data Engineer with 2–4 years of experience to design, build, and deploy data pipelines on Azure Databricks. You will ingest data from REST APIs and collaborate with senior engineers on real client projects, using Python and SQL to create scalable solutions.

You will work with AI coding tools to accelerate development, contribute to CI/CD and infrastructure as code, and engage clients directly to gather requirements and present progress.